I think the main problem is not setting isr.doApplyGains=True (required when isr.doFlat=False in order to get the different amplifiers on something approaching a common system). I built some new calibs (/projects/HSC/PFS/Subaru/CALIB-SuNSS-20220314), and the adjusted detectorMap fit now is:

reduceExposure.adjustDetectorMap INFO: Final fit: chi2=78975.644377 dof=49380 xRMS=0.040120 yRMS=0.061799 (0.005336 nm) xSoften=0.005633 ySoften=0.033382 from 24702/30239 lines

Below is an image showing the 2D sky subtraction I get at 876 nm. On the left is the vanilla pipeline (same command as above, plus isr.doApplyGains=True, and using the updated SuNSS calibs). The line complex at 876 nm is causing problems (you can see the awful horizontal artifacts). There are 10 lines in a 1 nm range, which can be resolved into two complexes of 5 lines each, each spanning about 0.05 nm. That range is comparable to the precision of our wavelength solution, so we're not getting good flux measurements. If I merge each of these two line complexes (using intensity-weighted mean wavelengths), I get the subtraction on the right. Clearly we're not getting all the flux, but I don't think we can do a better job until we get a better sky line flux model that doesn't treat all lines independently.

The other configuration change that could help this is modifying subtractSky2d.selectFibers.targetType to include only one of SUNSS_DIFFUSE and SUNSS_IMAGING. Oh, and we're obviously missing a lot of lines, including the O~2~ complex around 864 nm, but I don't think this is the place to fix that.

I'm a bit unclear about what the 2-D subtraction is doing. I thought that the line estimation code (resulting in arcLineSets) fits all the lines in a given fibre simultaneously, so I'd expect 2D sky subtraction to do the same thing, for the sky fibres, then use the resulting model to subtract sky from the other fibres. Apparently it's doing something different. Can you clarify for me?

The fluxes are measured by simultaneously fitting all lines that are blended. From the fluxes we derive a sky line flux model, which is currently done by averaging across fibers the fluxes of each line. It is the flux of each line from the sky line flux model that is used to subtract the sky lines from all fibers (that have a PSF model defined).

I believe that we are not measuring the line fluxes well because there are five lines separated by less than a pixel, so that small errors in the wavelength solution lead to large differences in the distribution of the measured fluxes of the blended lines for different fibers, which means that the sky line flux model is inaccurate. Maybe we could determine a reasonable sky line flux model if we knew the covariances between all the fluxes, but we do not record them.

Ah, OK, got it.

We may always have some (covariant) wavelength errors, and need to think about how to include them in the fitted sky model. Of course, if the wavelength errors are large enough to show up in the residuals we should be able to tweak the wavelength solutions, hence the use of the phrase "may always".

I think the best way to deal with this pair of line complexes at 876 nm would be to derive the sky line flux model from other lines: the OH lines should go up and down in concert, so we should be able to get the flux of each of the blended lines from the fluxes of the host of unblended lines.

In the long run, where we model the sky intensities from the sky fibres, that's definitely the thing to do (there are 8 vibrational occupations + probably a single rotational temperature, so a finite set of parameters).

However, we do still need to think about wavelength errors in the subtractions.

Following review:

Changes to drp_stella (changing default value for subtractSky2d.selectFibers.targetType to SUNSS_DIFFUSE) accepted.

Changes to obs_pfs (modifying the linelist) raised joint concerns, and as discussed in the comments above. Leave those as-is till a solution involving deriving the flux from lines outside the problematic area has been implemented.

From our weekly meeting: I will demonstrate that the line fluxes are being measured accurately in each fiber, proving that the problem lies in the interpolation of fluxes across fibers when there are huge degeneracies.

This image demonstrates that the line fluxes and aperture corrections are being measured correctly. The continuum is not being measured very well on both sides, around the 864nm bands, but the lines are being measured nicely.

I added some extra commits to clean up problems discovered while subtracting the lines. hassan, would you please check those too?

Agree with the additional commits described in https://github.com/Subaru-PFS/drp_stella/pull/255 . Only one very minor question on one of the commits, although this does not affect the code changes.

Addressed review comment and merged.