Reconsider how to handle fiberNorms

I wrote some notes in slack on how I think we need to do the normalisation, now that we've introduced fiberNorms, and Paul asked me to put them into a ticket. So here it is.

I shall assume that the fibre-to-fibre variation is achromatic – we know it doesn't seem to be from Arnaud's work, but let's add that complication later.

The procedure is:

• Using a quartz, measure the quartz spectrum in units of e/pixel; call this pfsArm.quartz. I believe that these are stored in the fiberProfile files.
• Extract the spectrum in units of e/pixel into pfsArm.flux

We now need to merge the arms. To do this, we

• Use the detectorMap to convert pfsArm.flux and pfsArm.quartz spectra to e/nm
• weight pfsArm.flux by pfsArm.quartz for each arm
• merge
  resulting in pfsMerged.flux spectra.

The quartz is used because it should be smooth (both within and across arms for a given fibre), but if the pfsArm.quartz spectra from different fibres are not identical (either in shape or amplitude) the pfsMerged.flux spectra from different fibres won't be on the same spectrophotometric system.
We therefore need to understand the relative shape and normalisation of the pfsMerged.quartz spectra (we don't currently calculate these while building fibre profiles, but we certainly could). Because for now we're assuming that the fibres have the same spectral response, we only have to scale each pfsMerged.flux spectra by a single constant-per-fibre, but the design of the fibreNorms allows us to use Chebyshev polynomials in wavelength. This constant (or later polynomial) allows for things like fibre throughputs, vignetting, and effective pixel size on the sky.
If we arrange for the mean of these constants to be 1.0, the pfsMerged.flux spectra will have units which are moderately close to e/nm.

Where do these constants come from? We want to end up with identical spectra measured by each fibre when observing a constant spectrum, constant surface brightness source (e.g. the sky, averaged over long enough periods of time). I'm not worrying about effects such as ghosting within the Popt2 corrector here – that will modify the detected surface brightness, and in fact it's that measured surface brightness that I'm taking to be uniform.

In theory, we could:

• Calculate pfsMerged.flux vectors for twilight flats
• Calculate the mean over a suitable wavelength interval, and correct for the known c. 1% illumination gradient (measured by taking data at a variety of instrument rotations)
• These means are the inverses of our desired corrections, the fiberNorms

Multiplying the per-fibre spectra of any on-sky constant surface brightness target (e.g. night sky) by the fiberNorms should result in identical spectra — which is what we need. The fiberNorms are the combination of many terms, and will have quite large amplitudes due to vignetting and fibre variations.

In practice it's easier to use the flat field screen rather than the sky, but that requires another step as the illumination isn't sufficiently uniform (we know that the JEG quartzes have a c. 4% gradient, which should be stable as they have a single lamp).

• Calculate pfsMerged.flux vectors for JEG flats
• Calculate the mean over the same wavelength interval as used for the twilight sky
• Correct for the known illumination gradient (measured by rotating the instrument rotator)
• Compare these values with the twilight fiberNorms calculated above, and save the corrections (c. 3%?), possibly by adding them to the fiberNorms files, maybe as a second HDU, or maybe in pfs_instdata. They do not need to be recalculated every run, unlike the "real" fiberNorms which are likely to change when we reconnect cables B and C.

We can then measure fiberNorms from JEG flats, and use them to monitor changes in the effective fibre throughputs.

The twilight-to-JEG ratios are needed in the construction of the fiberNorms, they they also allow us to monitor any evolution in the JEG gradients. There shouldn't be any, but "shouldn't" is no guarantee.

Does this make sense? It seems that the current code puts the fibre-to-fibre normalisation at some time (March 2024?) into the fiber profiles (referred to the twilight sky), and then puts the conversion from sky to JEG lamps into the fiberNorms; in otherwords, pfsArm.flux/pfsArm.norm*fiberNorm are JEG lamp normalised spectra. From something Paul said, this was done in response to something that I said in a meeting; I hope I wasn't clear rather than that I've changed my mind, making extra work for Paul.

When the fibre throughputs change (e.g. when we reconnect PFI to cable B), with this approach the change will be absorbed into the new fiberNorms so they are going to be really hard to interpret, being a mixture of illumination patterns and evolving fibre properties.

With this proposal, all the information about the normalisation of the profiles is in the fiberNorms, and only the shape of the response is in the profiles.