For PS1 and HSC filters:

PIPE2D-1427 defined the table for PS1, but I suggest to update PS1 cases 3 and 6 to above so that we applly the same inter/extrapolation to both PS1 and HSC.

Case 1: all arms are available and we use grizy. No inter/extrapolation needed.

case 2: b is missing and we use izy No inter/extrapolation needed.

case 3: n is missing and we use griz. No inter/extrapolation needed.

case 4: all arms in the medium resolution. Interpolate between the arms and use grizy.

The wavelength windows for the interpolation are:

fittingRangeMR = ((590.,610.),(610.,630.), (720.,740.),(740.,760.), (835.,855.),(855.,875.), (970.,990.),(990.,1010.))

Each of the gray shaded area in the plot here has two separate (but adjacent)  windows. For the interpolation, we can combine the two. For extrapolation (case 5), we fit a linear function to the two windows. Note the windows are narrower than my previous suggestion.

case 5: b is missing. Extrapolate to b from m just a little bit and interpolate between m and n. Use izy.

case 6: n is missing. Interpolate between  b and m. Use gri.

Now Gaia:

I made a change to case 5, where I now suggest to ignore and do not use for the focal plane fitting. Also, I do not use Rp in case 6 as it is a little scary.

Gaia filters extends shortwards of 380nm, which PFS does not cover. I suggest:

1. We use Gaia G and Bp responses truncated at 380nm (i.e., we ignore the responses at shorter wavelengths).
2. Estimate Gaia G and Bp magnitudes from the PFS spectra (let me denote them as G_truncated and Bp_truncated).
3. G_full = G_truncted + 0.017
4. Bp_full = Bp_truncated + (0.09967094 + 0.03244092·x + 0.04070014·x² - 0.03007215·x³ - 0.03245733·x⁴ + 0.02247528·x⁵), where x is g_HSC - r_HSC. I could have used PS1 instead of HSC.

We do this after the inter/extrapolations between the arms described below are applied.

case 1: all arms. We simply measure the synthetic fluxes in G_truncaed, Bp_truncated, Rp, g_HSC, and r_HSC. Then we compute G_full, Bp_full.

case 2: b is missing. We extrapolate r to the blue a little bit. The wavelength windows are:

fittingRangeLR = ((650.,670.),(670.,690.), (900.,920.),(920.,940.))

We linearly extrapolate first two windows (and use the remaining two windows in case 3).

case 3: n is missing. Extrapolate r to n a little bit and use G, Bp, and Rp.

case 4: all arms in medium resolution. Interpolate between b and m, and m and n. Use G, Bp, and Rp. The interpolation windows are:

fittingRangeMR = ((590.,610.),(610.,630.), (720.,740.),(740.,760.), (835.,855.),(855.,875.), (970.,990.),(990.,1010.))

These are the same windows as used for the HSC and PS1 filters for medium resolution.

case 6: n is missing. Interpolate between b and m, and extrapolate m to n. Use G and Bp. The extrapolation to n is a bit scary, but G should be fine as the extrapolation covers a limited part of the G-band. It covers a significant fraction of Rp, and let' not use Rp.

I hope my comments above are clear, but let me know, sogo.mineo . This is very complicated.

Thank you, sogo.mineo for implementing this feature. I think the code works OK. The left figure is a medium-resolution Gaia visit and the calibration accuracy is similar to PS1 visits in low resolution (I am a little concerned about the overall normalization offset in pfsSingle, but most FLUXSTDs have low-S/N in this visit and I would not worry too much about it). The 2nd is medium-resolution PS1 and is again similar.

Could you review this PR?

Merged. Thanks for reviewing.