The issue here is that fitContinuum.py needs to interpolate the wavelength data from the input detectormap, (ie detectormap.getWavelength() ). The wavelength information needs to be monotonically rising in order for the interpolation to function. The detectormap in this case is the output of bootstrapDetectorMap.py. 

Examining the wavelength information for this 'bootstrapped' detectormap, about 253 of the 600 fibers have wavelength information which is not monotonically rising. In each of those 253 cases, the non-monotonicity appears at the ends of the wavelength sequence, corresponding to the 0 < detY < 60 and 4160 < detY 4176 ranges in the wavelength direction, on the detector.

It seems that the Chebyshev2D modelling of the detectormap correction that is performed within BootstrapDetectorMapTask is correct. The problem lies in the spline modelling within DetectorMap. I suspect with no arc data close to the above detector ranges, the spline solution may not necessarily extrapolate to a monotonic solution.

The wavelengths information from the original July 2019 detectormap used as input to the bootstrap is monotonic for all fibers.

Note that the same error takes place if bootstrapDetectorMap.py is run twice (with the output of the first run replacing the July 2019 dataset as the default detectormap, ie. ingested into the butler calibration repository), with the same arc and field visits.

(Thanks to ncaplar for spotting this) the underlying cause of this problem is that the header information in the arc exposure in the example provided here (visit=436) is incorrect. It states that the arc is Neon, when in fact the lamp is a Krypton one.

The bootstrapDetectorMap.py loads only those reference lines for the lamp specified in the header and tries to match the observed lines against those reference lines. This lead to very few lines being matched (and being Ne lines, wrong matches) in the centre of the detector. The result is a very poor wavelength solution.

This was confirmed by making a local modification to bootstrapDetectorMap.py to force it to load only Krypton lines. The result was a good detectormap, with many matched lines and a monotonically rising wavelength solution, as expected.

reduceArc.py (with again a local modification forcing it to load only Kr lines) worked fine with a test with the same arc.

Additional tests will be performed with other arcs. It appears that there are other arcs incorrectly identified, but further checks need to be done to be sure.

If you have a list of visits with the wrong headers, we can put them into our database and get them fixed on ingest (PIPE2D-405).

That sounds like a good idea, and the correct procedure. Thanks, will do.

Plot https://pfspipe.ipmu.jp/jira/secure/attachment/12515/quiver-visit-436.png shows the difference between the observed and expected (from July 2019 detectormap) detector positions for matched lines during a bootstrapDetectorMap run. While there is an approx 6 pixel shift in the spatial direction, the transformation between expected and observed does not appear to be affine.

As ncaplar and J Gunn notes, if the fibers are separated across the individual CCDs, then there is separate affine solutions for each separate group of fibers. For example, from the plot, the rightmost 4 fibers show a systematic shift in both spatial and wavelength directions, whereas the leftmost 6 fibers show a spatial shift and a shear in the wavelength direction. Least squared fits on the two separate groups of fibers verify that the changes can be modeled by two separate affine matrices (with the matrix for the left group describing a 6 pixel shift and shear, and for the matrix for the right group a 5.5 pixel shift spatially and 1.5 pixel in the wavelength direction.

The current bootstrap model does not assume 2 separate CCDs. This should be considered.

Good point re pairs of CCDs (not true in the n arm). A full affine is probably overkill for the relative motion of the two CCDs, however Bernstein et al. PASP 129.7 2017 use a full affine for a similar problem and this might be appropriate here.

Note that we need an affine for the complete detector too, and the per-CCD term should generally be held fixed (except e.g. after warming the camera).

This needs to be added to the DetectorMap model not just the bootstrap (this is moot until we have a map that models all the fibres together).

Problem found to be incorrect lamp keyword values in header. With PIPE2D-536, this has been fixed. reduceArc.py no longer raises the problematic error.