import os
import numpy as np
from lsst.daf.persistence import Butler
from pfs.drp.stella import DetectorMap
root = "/scratch/pprice/pipe2d-669"
butler = Butler(os.path.join(root, "rerun/pipe2d-660/calib/brn/arc_brn/detectorMap"))
visit = 39
for arm in "brn":
    detMap = DetectorMap.readFits(os.path.join(root, f"rerun/pipe2d-660/calib/brn/arc_brn/detectorMap/DETECTORMAP/pfsDetectorMap-000039-{arm}1.fits"))
    #detMap = butler.get("detectorMap", visit=visit, arm=arm)
    lines = butler.get("arcLines", visit=visit, arm=arm)
    for fiberId in set(lines.fiberId):
        select = lines.fiberId == fiberId
        num = select.sum()
        fitWavelength = detMap.findWavelength(fiberId, lines.y[select])
        residual = lines.wavelength[select] - fitWavelength
        lq, median, uq = np.percentile(residual, (25.0, 50.0, 75.0))
        rms = 0.741*(uq - lq)
        print(arm, fiberId, num, median, rms)

Even with PIPE2D-660 (DifferentialDetectorMap), still getting median ~ -0.03 and rms ~ 0.01.

The detectorMap here is a SplinedDetectorMap, apparently coming from the calibs.
Using the DifferentialDetectorMap, the median goes to ~1e-3, but still have rms ~ 0.01.
We know the SplinedDetectorMap from the simulator is off by 0.5 pixel. For a dispersion of 0.07 nm/pixel, that corresponds to the observed 0.03 nm median offset.

The RMS of 0.01 nm appears to be constant, and probably not bad: it's 0.14 pixels. For arm=b, that's decent; we could probably set the limit at 0.02. For arm=r (0.09 nm/pixel), we're getting rms ~ 0.007 nm ~ 0.08 pixels, and for arm=n (0.08 nm/pixel), rms ~ 0.01 nm ~ 0.13 pixels. So the numbers look reasonable. Not sure how we got better than 0.01 nm previously, but that's our current state of the art.

But, why are we getting the SplinedDetectorMap, when we want the DifferentialDetectorMap? It's probably a butler thing, or maybe a simulator thing (unexpected times in the header).

>>> butler.get("detectorMap_filename", visit=57, arm="r")
['/scratch/pprice/pipe2d-669/CALIB/DETECTORMAP/pfsDetectorMap-000000-r1.fits']

sqlite> select * from detectorMap;
id|arm|ccd|spectrograph|calibDate|calibTime|visit0|validStart|validEnd
1|b|0|1|2020-11-26|2020-11-26T00:10:52|0|2020-11-25T12:05:26.000001|2025-10-31T00:10:52
2|r|1|1|2020-11-26|2020-11-26T00:08:25|0|2020-11-25T12:04:12.500001|2025-10-31T00:08:25
3|n|1|1|2020-11-26|2020-11-26T00:09:41|0|2020-11-25T12:04:50.500001|2025-10-31T00:09:41
4|m|2|1|2020-11-26|2020-11-26T00:12:08|0|2020-11-25T12:06:04.000001|2025-10-31T00:12:08
5|b|0|1|2020-11-25T23:27:34|2020-11-25|39|2015-12-22T00:00:00|2020-11-25T12:05:26
6|n|1|1|2020-11-25T23:26:26|2020-11-25|39|2015-12-22T00:00:00|2020-11-25T12:04:50.500000
7|r|1|1|2020-11-25T23:26:57|2020-11-25|39|2015-12-22T00:00:00|2020-11-25T12:04:12.500000
8|m|2|1|2020-11-25T23:26:38|2020-11-25|40|2015-12-22T00:00:00|2020-11-25T12:06:04

Strange: the SplinedDetectorMap is dated after the DifferentialDetectorMap, and the latter has calibDate and calibTime switched. This is probably because the simulator times can be a bit random (different images done in parallel, so times don't sort the same as visits).

Fixed the dates on the simulated data (both images and detectorMaps), and am now getting the right detectorMap in the test. Reverted the commit that loosened the precision, and actually tightened it a bit.