I have a bunch of changes associated with the handling of the blue arm from PIPE2D-316 awaiting the new review/merge policy.

I didn'tt expect Paul to need to change anything to detrend r/b as I thought that the data model was strong enough – but maybe his changes will reveal stupidity on my part.

I believe the data model was fine; it was just boring butler implementation stuff.

Here's a question from the ICS side: should we always indicate the normal/low res "arm" for bias and darks? For filenames, etc.

I don't think the pipeline cares, but that might make sense for humans.

The datamodel says:

(While theoretically we don't need to distinguish between the r and m chips, it's simpler to be
consistent, so we will have 16, not 12, biases/darks for the complete spectrograph)

fmadec, can you point me to some files that I can test on?

I am not sure I was clear enough last time, so let me (try to) clarify and be sure we are on the same page:

First, I am not using 2 arms at the same time (at least for now) but one at a time , so my point was that to be able to detrend the blue (but for the med that the same) while I previously detrented Red data, I cannot do it because it gave me an error (not able to find biases ...) so I had to create a separate drp folder to be able to process them.

so to reproduce that, make calib product for the red, process any red data, then make calib product for the blue and try to process either blue or red and that should not work anymore.

hopefully I was clear enough this time !

biases for blue: 12548..12562

arcs: 12593..12597

Using the current master branch of datamodel, obs_pfs and drp_stella, and the tickets/PIPE2D-359 branch of pfs_pipe2d, I ran:

pprice@tiger2-sumire:/tigress/pprice/pipe2d-378 $ mkdir CALIBS
pprice@tiger2-sumire:/tigress/pprice/pipe2d-378 $ ingestCalibs.py /tigress/HSC/PFS/LAM --calib CALIBS ../drp_stella_data/raw/detectorMap-sim-*.fits --validity 1000 --mode=copy
pprice@tiger2-sumire:/tigress/pprice/pipe2d-378 $ pfs_build_calibs.sh -r price/pipe2d-378 -c 10 -C /tigress/pprice/pipe2d-378/CALIBS -n -b visit=12548..12562 -d visit=12578..12592 -f visit=12623..12751 -F visit=12618..12622 -a visit=12593..12597 /tigress/HSC/PFS/LAM

This failed at the reduceArc stage:

lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:pfsConfig dataId:DataId(initialdata={'visit': 12595, 'dateObs': '2019-02-26', 'site': 'L', 'category': 'A', 'expId': 12595, 'arm': 'b', 'spectrograph': 1, 'field': 'ARC', 'ccd': 0, 'filter': 'b', 'expTime': 2.01, 'dataType': 'arc', 'taiObs': '2019-02-26', 'pfiDesignId': 4296015889, 'slitOffset': 0.0}, tag=set())   

This is because it couldn't find a pfsConfig for the exposure. I don't know what fibers were used, so I don't know what dummy pfsConfig to create. But I expect that if I created one, it would work fine.

I conclude that PIPE2D-316 fixed the problems with producing blue arm calibs. If you have any problems using the current master branch, please send me instructions on how to reproduce the problem (including a list of commands to run, and the error messages you get), and I will investigate further.

Does this data pre-date the header key that specifies the configuration?  If so, can we find some b + r data that allows Paul to check that this really does all work?

This doesn't appear to be an issue any more, and the pipeline has developed since it was filed. If it's still a problem, please open a new ticket with instructions on how to reproduce.