Wrote one. (But I don't know where to push it)

Can you initially add the python script to this ticket please?

I uploaded the python script forked from https://github.com/lsst-dm/generateCalibrations

Added the code to pfs_pipe2d and pushed branch tickets/PIPE2D-565.

I think this needs some more work to achieve what we need. This could supplant pfs_build_calibs.sh, but it's missing some important features.
I think it's important to update pfs_integration_test.sh to use this method (or provide a similar script) so that we can check that this works, and exercise it regularly.
The code needs to be edited for style (make sure it passes flake8 with the standard modifications) and docstrings added.

I think I have made all required changes, except for adding capability of specifying more keys for --id.
What keys can be used? I thought I could refer to the schema of table raw (below), but apparently those columns of double type cannot be used for --id.

CREATE TABLE raw (
    id integer primary key autoincrement,
    site text,
    category text,
    field text,
    visit int,
    ccd int,
    filter text,
    arm text,
    spectrograph int,
    dateObs text,
    expTime double,
    dataType text,
    taiObs text,
    pfsDesignId int,
    slitOffset double,
    dither double,
    shift double,
    focus double,
    lamps text,
    attenuator double,
    photodiode double,
)

Even the double fields can be used in the --id, e.g., --id field=FLAT exptime=10.0.

Thank you. I made all fields available in the YAML spec file.

I'm not sure hard-coding all the available keys is the right thing to do. The command-line syntax is free-form, and I think that's the way to go here too. For example:

    id: field=FLAT exptime=10.0

Are you concerned about spelling mistakes?

I see both advantages and disadvantages to your chosen implementation. Can you explain your reasoning, please?

I wanted to check the syntax of "value" in "--id key=value", needed to know the type of "value" in advance, and enumerated all key s with their types. But this is not essential. And I agree that writing "id: [key=value, key=value]" is clearer than mixing these "key=value" with other things like "config:..." and "validity:...". I am revising the code.

Currently, the output shell script is not a self-contained executable script but a mere function body. One has to source it (source out.sh) in order to execute it. Now that the shell script is written to a file, I think the shell script should be a self-contained executable script that begins with #!/bin/sh. Is it OK to make this change?

https://developer.lsst.io/work/flow.html#git-commit-organization-best-practices seems to say that I have to squash all commits made after the first review, rebase tickets/PIPE2D-565 onto the current master branch, and force-push the rebased tickets/PIPE2D-565 to github. Do I correctly understand the document?

Most important is that there is one functional change per commit. Ideally, there should be a one-to-one relationship between functional changes and commits (e.g., changes to the same code made before and after review get squashed together), but sometimes that is difficult to achieve, and so there may be multiple commits to achieve one overall functional change. But there should not be two functional changes in the same commit if you can at all help it. Does that make sense?

I think having the script be executable rather than sourced is a good idea.

I changed my code so that it will produce an executable script, and also added bootstrap functionality just now. Though I said in slack that I would provide lineList field for bootstrap, I decided not to provide it because it was not provided for reduceArc.py either.

I believe the overall code is ready for re-review. Could you check the new code? (I'm sorry for its big change from what it was on the first review)

I changed a few lines in order to promote --output parameter to a positional parameter, and to replace logger.info(f"message {s}") to logger.info("message %s", s). I squashed this change onto the existing commit and pushed it. I wrongly thought github would still be able to show diffs, but what I see now is only the new code of 1500 lines with no history. I should not have done this...

Ah, I found how to view diffs. Sorry for making a noise.

I learned just now that it is I who have to merge this ticket branch to master. The latest code (as I have pushed to github) is a little different from what Paul approved, but am I still allowed to merge this branch?

Please go ahead.

Thank you. I have merged this ticket branch to master.