How are the results to be delivered?

For row-windowed reads, I propose normal-sized PFSA files with the unread rows set to 0, and with some informative keywords. Or would you prefer something far different from regular PFSA files? Not sure what I'd do about binned images.

For row-windowed reads, I like the solution of setting unread rows to zero: with compression enabled we won't be wasting space, and this enables us to use the usual code without much in the way of changes.

I think binning in the spatial dimension should be ruled out, as any binning will confuse the fibers. I wonder what we could gain from binning in the spectral dimension that we can't achieve through windowing. The only thing I can think of is spectral coverage, but I can't imagine using arcs for this purpose rather than a continuum source. I therefore suggest we not consider any form of binning.

But I forgot to say: my original question was regarding the format for the results from the pipeline (presumably a dict mapping fiberIds to flux measurements). Who/what needs the results, and what format would be most suitable?

Jim-sensei believes that we'll need to bin. I'm not sure that he's right in this case, and am happy to postpone that part of the work.

Why is this still in the sprint? I think we decided not to implement binning

Agreed. Removing that ticket from the eng run sprint.

When reading a band of rows, W_CDROW0 and W_CDROWN are added to the PHDU. All pixels outside that band are set to 0. Other that they, they are normal PFSA files. Will try getting some from SuNSS.

cloomis will provide price test data by tomorrow (Thu) evening for Paul to check.

Example file ingested on tiger. visit=69294 arm=r. SuNSS within the dome; am guessing with dome lights.

reduceExposure.py kind of worked on the test data, but there were a few problems:
1. The speed was much too slow (80 sec, even with doAdjustDetectorMap=False).
2. The unread area was not masked.
3. The good area was on a pedestal of about 1400 counts.

To fix:
1. Added a switch to reduceExposure to disable emission line photometry (doMeasureLines=False), which reduces the runtime down to about 14 sec.
2. Added code in ISR to mask zero pixels. This is done before overscan subtraction, so the bad pixels can be excluded from the overscan. Slightly annoyingly, this requires masking the pixels SAT (mask value for overscans is hard-wired). I added NO_DATA to make it a bit clearer what's going on.
3. The great reduction in overscan real estate means we need to use a different overscan method, like overscanFitType=MEDIAN.

I recommend a command-line something like:

reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/ --rerun price/pipe2d-914 --id visit=69294 -c isr.doFlat=False doAdjustDetectorMap=False doMeasureLines=False isr.overscanFitType=MEDIAN

We know that these data are going to need these tweaks. Can we look at the metadata and set them automatically?

The detectorMap adjustment can be done (at least, in the red; wouldn't try it in the blue) at the cost of about 20 sec (total runtime 36 sec):

reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/ --rerun price/pipe2d-914 --id visit=69294 -c doWriteCalexp=True isr.doFlat=False doAdjustDetectorMap=True doMeasureLines=False isr.overscanFitType=MEDIAN centroidLines.doSubtractContinuum=False

I don't know how we could possibly apply the adjustment by looking at the header.

I meant adjustment to the configs. So that a simple reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/ --rerun price/pipe2d-914 --id visit=69294 would work

We could have the detectorMap adjustment and line measurement also respond to the presence of header keywords, but isr.overscanFitType is used deep in a massive LSST function, so overriding that would be real messy. We could do it, but blrgh!

Sorry to add another request, but after reviewing the code, I would have thought it better to apply masks based on the (inverse to the) region defined by the W_CDROW0 and W_CDROWN keywords. See earlier comment by cloomis.

I think that the approach I've adopted (masking zero-valued pixels in the raw) is superior. It means I don't have to muck around with xy0 on the raw, and it catches problems in the full readouts as well.

Back to configs. So the problem is that the configs are frozen before we read the metadata? That's true. So I think I'd add an option for "this sort of data" and have that set your magic values. Would that work?

Why don't we just add a configuration file to be used for "this sort of data"?

a `-C` file in a well-known place? Is that really cleaner?

I don't know how to make it work cleanly otherwise.

Can't you set values in the config validate function?

We could tweak the ISR configuration based on a reduceExposure configuration, but not a header keyword.

That was what my 'an option for "this sort of data" and have that set your magic values' proposal was supposed to mean.

OK, here 'tis:

reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/ --rerun price/pipe2d-914 --id visit=69294 -c isr.doFlat=False quickLook=True

I'm sorry, but quicklook isn't correct – that implies e.g. cheap sky subtraction. This is closer to windowed

Renamed to windowed.

Merged.