How does this relate to PIPE2D-1194?

That one says "based on lab flats". It may not be necessary once these are available, but I don't know the true goal of that ticket (looks like something to do with persistence).

From my notes:

Running the following with PIPE2D-1205:

constructFiberFlat.py /work/drp --calib=/work/price/CALIB-202304-eng --rerun=price/pipe2d-1221 --id visit=91203..91235:2^91292..91339:2 arm=n spectrograph=1 --cores 50 -c profiles.profileSwath=2000 profiles.profileOversample=5 minSNR=20 profiles.pruneMinLength=3000 profiles.associationDepth=100

Having problems with transactions, and then it can't find a dark.
There's a dark with a bad name: pfsDark-2023-04-19T16:32:16.617-000000-n1.fits

Let's build a new dark.

(lsst-scipipe-3.0.0) price@pfsa-usr02-gb:/work/price/CALIB-202304-eng/DARK $ rm pfsDark-2023-04-19T16\:32\:16.617-000000-n1.fits
sqlite> DELETE FROM dark WHERE arm = "n";
sqlite> DELETE FROM dark_visit WHERE arm = "n";

constructPfsDark.py /work/drp --calib /work/price/CALIB-202304-eng --rerun price/pipe2d-1221 --id visit=90792..90797 arm=n --batch-type=none

That worked, and looks like the name is correct.

ingestPfsCalibs.py /work/drp --calib /work/price/CALIB-202304-eng /work/drp/rerun/price/pipe2d-1221/DARK/pfsDark-2023-04-19-090792-n1.fits --validity 1000 --mode=copy -c clobber=True

Back to the flat.
Final result looks good.

sqlite> DELETE FROM flat WHERE arm = "n";
sqlite> DELETE FROM flat_visit WHERE arm = "n";

ingestPfsCalibs.py /work/drp --calib /work/price/CALIB-202304-eng /work/drp/rerun/price/pipe2d-1221/FLAT/pfsFlat-2023-04-23-091203-n1.fits --validity 1000 --mode=copy -c clobber=True

The new n1 flat doesn’t serve to flatten the data: the “flattened” data still has strong detector features visible. I’m thinking that this must be the result of persistence, but perhaps I’ve missed something?
In the below plot, I’ve taken two points on the detector (one is deliberately selected to be on a strong positive feature in the flat), separated by 6 pixels (about one fiber) horizontally. I’ve found fiber traces on the image, and then used the traces and extracted spectra to create a model for the image. I’ve divided the image by the model to give a normalised image, and then plotted the mean and ratio of the values on the normalised image at the two points of interest. We have multiple images as the slit is dithered, giving many points, but there is not a consistent ratio of flux between the two points (despite the fixed distance between them in the spatial dimension). The black cross is the corresponding point for the flat-field image.

Can we see some images too, please?

In the below image, the unflattened image is shown on the left, and the flattened image is on the right.

That's depressing. What does the flat look like?

What's the ratio of the pre- and post-flattened images? The flat looks fine, so I don't understand why it didn't work better. With proper handling of 0/0

The ratio of the pre- and post-flattened images is the flat.

If you look carefully, the flat has removed high-frequency structure. The bright features are smeared out in the post-flattened image.

Yes, the ratio should be the flat modulo bugs. However, something is wrong. If you take two quartz exposures and divide one by the other you should get something close to unity; right? Your findings suggest that the dithered flat code isn't equivalent to that procedure.

The problem was due to the normalisation of the individual exposures, wherein we measure the fiber profile, extract spectra, and produce a model image which we use to normalise the exposure. This process naturally samples things nicely in the spatial dimension (through the combination of reducing every row of every fiber to a single number, and then dithering). The problem is that because we have been dividing by the actual spectra, the normalised image didn't contain any structure in the spectral dimension. I've been trying to deal with that by applying a median filter to each spectrum, and I like the results I get with a median filter of half-width 400 pixels. rhl has suggested we normalise by an average spectrum, which will require future developement (PIPE2D-1226).

I have installed the flats in CALIB-2023-04-v2.