• 115066r4 has over-subtracted sky lines at x=1950, y=1813 --> fiberId ~ 2302, wavelength ~ 782

  There are no CRs or bad data needing to be rejected, but on the edge of the line a lot of the samples were piling up on one side of the bin. The x value used for the bin was the center of the bin, and so the spline was not going through the points, and points were getting rejected. Changing the x value to the median for the bin results in a much better result.
  
  Before, default parameters:
  
  mergeArms.fitSkyModel INFO: Fit focal plane function: chi^2=399679.493485 length=4176/4176 numSamples=101 numGood=405065 numBad=7838 numRejected=9929
  
  Changing the x value to the median for the bin:
  
  mergeArms.fitSkyModel INFO: Fit focal plane function: chi^2=407918.079300 length=4176/4176 numSamples=101 numGood=404022 numBad=7492 numRejected=11267
  
  Fixed and blockSize=10 oversample=1.5:
  
  mergeArms.fitSkyModel INFO: Fit focal plane function: chi^2=420375.766988 length=4176/4176 numSamples=101 numGood=417600 numBad=2563 numRejected=1656
  
  Results from the last look best. Note the large reduction in the number of rejected points.
  

• Horizontal bands in all-spectra plot of 115066

  fiberId=922 is under-subtracted, 929 is over-subtracted.
  blackSpotDistance is 1.5 and 1.9, respectively (black spot penumbrae have a radius of ~ 1.108).
  Plotting the normalized pfsArm, 922 appears higher and 929 is lower, but it's subtle.
  fiberId=930 is a FLUXSTD with r ~ 17.5 mag: but if the problem was leakage from the neighbouring fiber then we would expect 929 to be higher, not lower.
  The block includes fiberId=[912, 914, 922, 928, 929, 939, 945, 948, 949].
  As far as I can tell, the effect is in the data.
  Clean wavelength region: 710-712. Over this region, fiberId=922 is 4.3 sigma high, 929 is 4.3 sigma low.
  (sigma is 1e-4 of the normalized flux.)
  
  The quartz taken immediately before (115064) is bmn. I'll look at 115068.
  It shows fiberId=922 has normalized flux over 710-712 as 1.055, while all the other fibers are 0.975 +/- 0.019.
  fiberId=929 is slightly high (0.99), but not obviously so.
  I conclude that the horizontal bands are due to fiber transmission changes.
  

• There seems to be a difference between the sky flux in the pfsArm and pfsMerged files: the sky-subtracted pfsMerged files aren't zero, but the pfsArm files are. 115029b4, fiberId=2375

  reduceExposure.py /work/drp --calib /work/drp/CALIB --rerun /work/drp/rerun/kiyoyabe/erun/run18/20240915:price/pipe2d-1553 --id visit=115029 arm=b spectrograph=4 -c useCalexp=True
  mergeArms.py /work/drp --calib /work/drp/CALIB --rerun /work/drp/rerun/kiyoyabe/erun/run18/20240915:price/pipe2d-1553 --id visit=115029 -c fitSkyModel.blockSize=20
  
  I can't reproduce the problem. There appears to be a small wavelength offset at the 5577 line, but I see it in both the pfsArm and pfsMerged spectra. I rebuilt the b4 detectorMap with the latest code (the current detectorMap is the old MultipleDistortionsDetectorMap). The NaI lines at 589 nm seem to always have bad residuals, and explicitly removing them from the line list makes the fit a bit better and then the sky subtraction is a bit better. But I'm going to leave this until RHL gets back with more detailed information.
  

I'm continuing to look at the horizontal bands in the all-spectra plot, which appear to be mainly due to transmission changes in the fibers. I'm attempting to measure the transmission changes using sky lines.
The FindLineTask includes methods to perform a Gaussian fit to lines, fitting for the amplitude, width, center and a linear continuum. Getting a decent selection of lines is important; in the following plot, I show various histograms for the lines, with the quartiles shown as red lines.

After selecting lines, I normalize them by the median flux of the line across all fibers. It's clear that some fibers are systematically high or low relative to the median.

I thought to fit a 2D polynomial as a function of position on the camera (for aperture corrections) and a low-order function as a function of cobra position (for variation on the sky), but it's not clear that those are really necessary. Simply taking a weighted average for each fiber does a pretty decent job.


Applying these tweaks to the line fluxes smooths out the per-fiber variations.

Applying these tweaks before the sky subtraction (using blockSize=10 oversample=1.5) yields a much better result (stdev 77 --> 47). Here is the original (with the same sky subtraction parameters) and the tweaked sky spectra after subtraction:

See also https://sumire-pfs.slack.com/archives/C3AU3VCMU/p1729844167180929 ( )

Important fixes were moved to PIPE2D-1586. Jared is taking over the parameter tuning.