So I looked at this today, I've looked at the flat used to build fiberProfiles.
As Paul pointed out on slack, you can also see those sawtooth features in the extracted spectra :

When I do the extraction using the same detectorMap and my naive box-car code, I don't see those features, (my profiles are 5 pixels wide).

So yes I think there is an issue in the fiberProfiles, I would try to regenerate wider profiles first.

I don't understand your conclusion. The normalisation should recover the input spectra used to estimate it in the first place, so if it doesn't I'd have thought that that meant that there was a bug in the code.

The previous plot just show that we're missing some flux and that's coming from the fiberProfiles.
But actually in the range(1000, 1050 nm) that yabe-san were showing the extraction of the quartz isn't terrible.

Looking at the pfsArm file, if you compare pfsArm.flux and pfsArm.norm

Those two should basically match so I don't understand from where the normalisations come, I mean they are embedded in the profiles, so I know their provenance but I don't understand the numbers inside, maybe the profiles were generated using some old code ? It's puzzling.

Same plot with the merged file, where we can see the smooth normalisation.

So I've regenerated the profiles, and re-reduced the data, now the normalisation in the pfsArm makes sense :

But the merged spectra is wrong (I withdraw that statement)

With the new profile the sawtooth feature did disappear

Full wavelength range :

I've also looked at 91401, which is another quartz taken on all 3 arms, (90845, used for the profiles generation is n-arm only)

We expect ~=unity, it looks actually worse in the n band.

Looking a bit closely at the mergeArms.py code, it seems to be there is a bug indeed, in normalizeSpectra function.

for ss in spectra:
            badBitmask = ss.flags.get(*self.config.mask)
            for ii in range(len(ss)):
                dispersion = calculateDispersion(ss.wavelength[ii])
                nn = interpolateFlux(ss.wavelength[ii], ss.norm[ii]/dispersion, wavelength, fill=0.0)

interpolateFlux already applies the jacobian by default, so it looks like it's applied twice.

The n1 fiber profiles were created with the following command:

constructFiberProfiles.py /work/drp --calib=/work/price/CALIB-2023-04-v1 --rerun=price/pipe2d-1221 --id visit=91014 arm=n spectrograph=1 --config isr.doFlat=True profiles.profileRadius=2 profiles.profileSwath=2000 profiles.profileOversample=5 --batch-type=none --no-versions

It turns out that the profiles for n1 are just plain awful:

I've no idea how that happened, but re-running the constructFiberProfiles command, I get much better profiles:

And the normalisation is much better too:

I'm going to create a new CALIB directory with these profiles.

Built a new calib directory:

price@pfsa-usr02-gb:/work/drp $ cp -r CALIB-2023-04-v2 CALIB-2023-04-v3
price@pfsa-usr02-gb:/work/drp/CALIB-2023-04-v3/FIBERPROFILES $ rm pfsFiberProfiles-2023-04-20-090845-n1.fits 
price@pfsa-usr02-gb:/work/drp/CALIB-2023-04-v3 $ sqlite3 calibRegistry.sqlite3 'DELETE FROM fiberProfiles WHERE arm = "n"; DELETE FROM fiberProfiles_visit WHERE arm = "n";'

ingestPfsCalibs.py /work/drp --calib /work/drp/CALIB-2023-04-v3 /work/drp/rerun/price/pipe2d-1242/FIBERPROFILES/pfsFiberProfiles-2023-04-21-091014-n1.fits --validity 10000 -c clobber=True --mode=copy

price@pfsa-usr02-gb:/work/drp $ ln -s CALIB-2023-04-v3 CALIB

Looks like another place where even a simple QA process would be very helpful.

The conversation about the code snippet arnaud.lefur pointed out has been moved to PIPE2D-1248.

The entire process is still running but I checked that visit, which has been finished, that particular feature is gone now.

Merged to master a small fix to FiberProfileSet.plot.