The large variation is due, at least in part, to bad fiber positions. I looked at the ratio between the synthetic i-band magnitude of a FLUXSTD from pfsMerged vs. its PS1 i-band flux. The first plot below is a good visit; the ratio is roughly constant across the focal plane. The 2nd plot is a different visit but with the same pfsDesign. FLUXSTDs in the right half of the field show significant flux losses. The left half is not too bad. This sort of bad visits contributed to the normalization scatter above.

I looked into this bad visit a little further and found it is unlikely due to bad guiding. The figure below is from Moritani-san.

There might be some scale/rotation offset, but an offset should introduce a spatial pattern that smoothly varies across the field. The bad visit above seems to show a sharp boundary around the center. Is it due to bad fiber positions, then? That seems unlikely.

The figure above shows the distance between pfiCenter and pfiNominal for each FLUXSTD in units of um. Most stars have an offset less than 10um, and there is no global pattern (i.e., no sharp boundary at the center). Another way to look at this is to plot the flux loss as a fn of fiber offset as shown below.

The flux loss is defined in the same way as in one of the above plots (magMergedDiff...png) and the fiber offest from pfsCofig is on the horizontal axis. There seems to be two horizontal sequences, which correspond to the left/right half of the field of view. I do not know what caused this flux loss, but the smooth focal plane function that we use in fluxCal cannot fix such a strong/sharp spatial structure. A practical solution for now is to remove those bad visits from coadds. Even after excluding the bad visits, I still see a normalization variation of about several per cent for the same FLUXSTD. This has to be understood.

This is a different visit from the example in the previous comment, but the symptom is the same; the flux loss is significant and has a strong spatial gradient. The focal plane fit is not more stable thanks, I think, to PIPE2D-1558. It is not perfect yet, but now the observed i-band magnitude scatter is comparable to the color scatter, which means that I am in a position to start tweaking the flux calibration vector along wavelength (which is a feature disabled until now). See PIPE2D-1579 for the explanation of the figures.

There are two separate effects. One is a hardware issue and is illustrated in the plot above (visit=114914), where most stars are missing fluxes. This is being investigated in PIPE2D-1580, and the rotator may be doing something bad. The other is that a fraction of FLUXSTDs are missing fluxes, although they do not show a coherent structure over the focal plane. They are due to significant fiber offsets; these fibers did not converge. INSTRM-2381 addresses this and these fibers will be given NOTCONVERGED fiber status. However, this will be applied to data taken after October 2024 and all the previous data do not have this mask bit. This was the confusion I had. These unconverged fibers thus have to be identified manually in data taken before Oct 2024. Unconverged fibers show a significant flux loss and I've found that the S/N cut in PIPE2D-1592 effectively identifies and removes these fibers.

Even after excluding the bad visits and unconverged fibers, there is still a lot of room to improve fluxCal because we have not optimized the fluxCal parameters yet. After a lot of experiments, I propose to

• exclude the y-band from the focal plane fit as it suffers from the n-arm.
• enable fluxCal vector to vary along wavelength (i.e., fitFluxCal:fitFocalPlane.polyWavelengthDependent=True): this reduces the color residual and overall i-band residual as well.
• add the systematic error to PS1: Currently we do not include systematic error to the PS1 photometry if I am not mistaken. But, PS1 photometry has a systematic uncertainty and it is larger than a random uncertainty for bright sources like FLUXSTDs. I confirmed that, if we add 1% flux uncertainty in the quadrature (i.e., flux_error_total^2 = flux_error^2 + (flux * 0.01)^2), fluxCal improves.
• reduce the focal plane fit tolerance to 1e-3: the current tolerance is 1e-6, but this is too strict if we add the PS1 error above. Experiments have shown 1e-3 is reasonable (fitFluxCal:minimizationTolerance=1e-3). No major change from 1e4 and the result gets worse if 1e-2.
• increase the polynomial order to fit the focal plane from 3 to 5: many visits have significant spatial structure and we can/should increase the polynomial order to 5. A major drawback is that the compute time is much longer; it will take ~1 hour. I am sure that people will not be happy, though...

These plots summarize the changes. The first one is the current default (no wavelength dependent correction, 3rd order polynomial, etc), and the 2nd one includes the suggested changes above.  Both the i-band difference and g-z color difference from PS1 (two panels on the bottom left) are improved.

Figure caption:

There is still a several percent scatter. I think some (or maybe majority) of which is due to random fiber offset (the amount of fiber offset differs from fiber to fiber and there is not global structure over the focal plane). The random scatter cannot be corrected for by the focal plane fit. I will update my write-up on the fiber loss in PIPE2D-1581.

Could you review this pull request?

Merged. Thanks for reviewing the PR.