I spent some time this week writing a function to do simultaneous PSF fitting of all lines, and then made improvements to make it run within a reasonable amount of time. Now I have a catalog of aperture (5 pixel radius, summed pixels within aperture and scaled up to appropriate radius) and both unforced (at centroid position) and forced (at detectorMap position) PSF (a DoubleGaussianPsf with FWHM=2.5 pixels) flux measurements. Let's play with it.

Above I show the same set of lines (690 < wavelength < 696, 24 <= fiberId <= 38) that rhl showed in his plots (he showed fake spectra from the line fluxes, but I'm just showing the line fluxes). Each flavor of photometry (two of the flavors have a wavelength offset, for clarity) seems to be consistent (modulo an aperture correction between the aperture and PSF fluxes), but the flux measurements for the different lines aren't very consistent, and sometimes the pattern of line fluxes is different between lines (e.g., the center line has cyan as the brightest, but the next line to the blue has dark blue as the brightest; the differences are greater than the errors, which are a few percent). I was concerned that forced photometry might get bad results due to a noisy wavelength solution, but for the spread of centroid shifts that we see between the forced and unforced (less than 0.5 pixel, and typically around 0.2) results in flux differences of a percent or two, but the effect is larger than that.

Above, I've divided by the fiberTrace normalisation (median filtered with a window of 9 pixels to reduce pixel-to-pixel noise), which makes the fibers more consistent, but doesn't solve the problem of patterns (e.g., the center line has cyan as the brightest, but the next line to the blue has cyan as the faintest).

Plotting all fibers in this diagram makes a mess, so I spread the fibers out in the wavelength direction, and found a consistent pattern. I think this is due to an illumination difference between the left-hand and right-hand sides of the spectrograph in the fiberProfiles.

Taking the line at 692.51295 as the reference, I plotted the ratio of the line flux in each fiber to the average line flux across fibers. The noise appears to be at the level of a few percent, which is in reach of effects previously noted. However, there is a clear difference for the red points (690.2737) at low fiberId values. I've no idea where this could be coming from.

import pandas
from astropy.table import Table
df = Table.read("photometry.fits").to_pandas()
avgFlux = df[["wavelength", "psfFlux_forced", "psfFluxErr_forced"]].join(df.groupby("wavelength")["psfFlux_forced"].mean().rename("meanFlux"), "wavelength", "inner")
relFlux = avgFlux["psfFlux_forced"]/avgFlux["meanFlux"]
relFluxErr = avgFlux["psfFluxErr_forced"]/avgFlux["meanFlux"]
pattern = relFlux[df["wavelength"] == 692.51295].to_numpy()

import matplotlib.pyplot as plt
for wl, col in zip((690.2737, 691.45295, 694.14355, 695.0961), ("rgbc")):
    select = df["wavelength"] == wl
    plt.scatter(df["fiberId"][select], relFlux[select]/pattern, color=col, marker="o", alpha=0.2)

plt.xlabel("fiberId")
plt.ylabel("Flux ratio")
plt.show()

While also attending the MOS workshop this week, I did some eigenvector analysis of the measured line fluxes. In what follows, I use forced simultaneous PSF-fitting fluxes from the continuum-subtracted 2D spectra. The continuum subtraction is decent (not good, not awful): in general it's more over-subtracted than under-subtracted, and there's a slight misalignment that's more noticeable in the blue (maybe that's just where the continuum is bright). That's consistent with the line-subtracted image, where the lines almost all look like a backwards "C".

The first normalised eigenvalue is 93%, then 3, 2 and 1%, which is somewhat reassuring. Here are the top six eigenvectors. The zeroth is the average, as usual, and looks reasonable (apart from a few strange values, which I think are from small measurement problems). The next three scale up and down various line combinations (I wonder if we can extract some physics from this when we get a better handle on the measurements?). Not entirely sure what the fourth and fifth are.

Fitting the measurements with the top six eigenvectors yields the following coefficients. Red points are diffuse fibers, while blue are imaging; there's no great difference between those until we get to the third coefficient (bottom-left), where there's a clear discontinuity between the red and blue points. My guess is that this has to do with a connector. I think this is the same phenomenon observed in the flux ratios plot in the previous comment, as the 690.2737 line (which is red in that plot) is close to zero in the third eigenvector while the other lines in that plot have negative values.

I think this shows promise for modeling the sky line fluxes for the 2D sky subtraction module, but we're going to need better line measurements. I need to look at improving the detectorMaps and check that the fiberProfiles are reasonable.

There are/should be (at least) three things going on with the line ratios:

• The occupation of the vibrational levels changes.  These are the (quantum, not pca!) eigenstates which determine the relative intensities of the bands
• The rotational temperatures change.  This changes the intensities within the bands
• The rotational temperature is a function of position.  This will show up in imaging but not diffuse fibres

Next time you make the PCA plots can you scale the eigen vectors by sqrt(eigenvalue)?  that way you get an idea of the relative importance of the terms.  Also, colour code the way that drp/stella/utils does (red for diffuse, green for imaging)?

I assume that "eigenflux" means eigenvector (it could be interpreted to mean the coefficients).  Let's stick with the standard names!

I don't understand why the diffuse fibres, with c. 72% of the throughput according to the fiberTraces, have roughly the same 0th component.

What's going on with component 1 (top middle)?  It looks triple-valued to me.  The only thing I can think of is something about the pattern of gaps along the slit (usually 2 or 3 but 1 in the large diffuse block), but I don't see how that would work.

Component 2 is also weird; small for almost all fibres.  Component 3 even more so.  If I had to guess, I'd say it was something in the flux estimation, probably the continuum.

do simultaneous PSF fitting of all lines, and then made improvements to make it run within a reasonable amount of time

Is this all lines within one spectrum, or all lines in the spectrograph?  I assume the former.  I wouldn't expect this to be slow, it's no worse than independent measurements followed by a least-squares solver (N_line^2 time, or N_line^3 if you wanted the covariances).

Here's an image of the residuals after continuum subtraction and subtraction of the PSF fits (with improved detectorMap and fiberProfiles). The green circles indicate the extent of the six fibers x 5 wavelengths originally highlighted in this ticket.

Are you using a different sky model in each fibre (I suspect that the answer's yes)?  I think we need to use the same model for all fibres, and that may be part of the reason for inconsistent line ratios between the fibres.   The fiberProfile alignment looks good, but we're oversubtracting sky.

I think this is now in a position to merge and close. I believe the problems were due to a combination of the following:
1. The photometry was not normalised by the fiber throughput. This shouldn't have greatly affected the relative line strengths of lines close in wavelength, but it's definitely something that needed fixing.
2. The photometry algorithm was a bit naive: it was subject to CRs and didn't handle blends. This wasn't a huge contributor to the issue of changing relative line strengths, but something I addressed since we'll need it for 2D sky subtraction.
3. The continuum fit wasn't masking known emission lines, which means it could be picking up the wings of lines and therefore we were oversubtracting the continuum.
4. The detectorMap and fiberProfiles were inaccurate. A bad detectorMap means that the line centers aren't in the right place and that the fiberTraces used for continuum subtraction are wrong. Note that there is a strong relationship between the detectorMap and fiberProfiles (construction of the profiles requires us to know where the traces are, and the profiles are used for continuum subtraction during arc/sky line centroiding when fitting a detectorMap), so some iteration is necessary to bootstrap a good solution. I recommend the following procedure when starting from absolute scratch: constructPfsBias, constructPfsDark, constructFiberFlat, bootstrapDetectorMap, constructFiberProfiles, reduceArc, constructFiberProfiles, reduceArc (with the usual ingestPfsCalibs between each).
5. Ignoring/miscalculating statistical errors (I discovered multiple errors in my photometry error calculations).

With the above addressed, I believe that we're down in the noise. As a demonstration, I offer photometry from 20 contiguous SuNSS exposures. The following plot shows the flux of the 692.51295 sky line for the same group of fibers RHL chose earlier (24 <= fiberId <= 38) as a function of visit, with the fibers color-coded. Firstly, the pattern of colors differs from visit to visit. Secondly, the error bars greatly overlap, showing that the data is generally consistent within the statistical errors.

This looks much better.  I agree that we should close this out and try using the improved fluxes.   If we find problems (and we will!), let's open a new ticket.

Changes look fine to me. Maybe write out the adjusted detectormap for debug purposes? But that can be a separate ticket (if not already supported).

Thank you, both.

Merged.