I am unable to reproduce the plots that Satoshi Hamano showed last night. His plots had a solid red band extending all the way down the right-hand side, indicating that fibers were not being corrected. The following plot looks very different: no fiber is all red or all blue (in line with our expectations after applying slit offsets in the detectorMap), and we're seeing features that we know are present in the data (like the epoxy bumps on the edges of each CCD) and revealing large-scale tree rings clearly.

Satoshi Hamano, could you please provide examples that demonstrate the problems you've discovered, including the dataId and plots for a small number of egregious cases?

import matplotlib.pyplot as plt
import matplotlib.cm
from matplotlib.colors import Normalize
from pfs.drp.stella.fitDistortedDetectorMap import addColorbar

dataId = dict(visit=98626, arm="r", spectrograph=3)
lines = butler.get("arcLines", dataId)
detectorMap = butler.get("detectorMap_used", dataId)

dx = lines.x - detectorMap.getXCenter(lines.fiberId, lines.y)

norm = Normalize(vmin=-0.1, vmax=0.1)
cmap = matplotlib.cm.get_cmap("coolwarm")

fig, axes = plt.subplots()
axes.hexbin(lines.x, lines.y, C=dx, gridsize=100, cmap=cmap, norm=norm)
addColorbar(fig, axes, cmap, norm, "dx (pixels)")
axes.set_aspect("equal")
axes.set_xlabel("x (pixels)")
axes.set_ylabel("y (pixels)")
axes.set_title(f"{dataId}")

Have you chosen a mod4 visit intentionally ? Same plot with full density quartz, it certainly feels different :

As an example, dataId={visit=98627, spectrograph=3, arm='r'}.

I make an image from the pfsArm and traces as follows. I measure the profiles center and widths from the calexp.image.array and image.array.

from pfs.drp.stella import SpectrumSet

calexp = butler.get('calexp', dataId)
detMap = butler.get('detectorMap_used', dataId)
profiles = butler.get("fiberProfiles", dataId)
pfsArm = butler.get("pfsArm", dataId)

spectra = SpectrumSet.fromPfsArm(pfsArm)
traces = profiles.makeFiberTracesFromDetectorMap(detMap)
image = spectra.makeImage(calexp.getDimensions(), traces)

98627 is a different visit than what I've posted, but I don't think it can look so very different. So I believe your results are different from what both Wilfred Gee and I get (although both of these are based on the arcLines, but I think those are generally regarded as reliable), which I think means you need to validate your work before we can proceed on this.

This is the same request as the one on PIPE2D-1330. Please provide the code as .py files on GitHub, and also details of how your centroids compare with the ones in the arcLines files. It's perfectly possible that your ones are correct, but we need to sort out the differences (and potentially correct the centroids that the pipeline generates).

I checked my code step by step, but couldn't identify the cause of the large X offset seen in my r3 plots. I attached a notebook that contains the part of calculating residual image in my extraction QA script. I run the notebook with w.2023.45 version, and confirmed that the same large X offset could be seen in the residual image. Could you check if there is a problem with my code?
largeXOffset.ipynb

What in the notebook looks reasonable, but it's far from enough to be able to reproduce your results.

I attached the reproduced result. largeXOffset-2.ipynb

The python code used for generating the QA plots is also attached, but the readability is poor. extractionQa.py

Satoshi Hamano is measuring the x position of the trace in the calexp and in the model image (constructed from the pfsArm, profiles and detectorMap_used), and his plots show that there is a difference in these x positions for those fibers on the far right. This is also revealed in a straight subtraction of the calexp and the model image. These differences are not present in the arcLines measurements written by the pipeline (which measures the position of the trace in the calexp) because the problem is in the model image: the profile is off-center (by about a third of a pixel) for those far-right fibers. This is further fallout from the inaccurate detectorMaps caused by the flat-field screen not being used on the first night of the July run. Below I show the profile for fiberId=1307. The results of profiles[1307].calculateStatistics() demonstrate the problem also (note the centroid values):

Struct(centroid=[-0.3664167395365968 -0.33592957113018446 -0.2319049640704939]; width=[2.0435098090603625 2.0388422530999777 2.073611949172245]; min=[-0.006259117354348317 -0.002724690172294439 -0.0007977413487588208]; max=[2.3501915455243494 2.3594814341754806 2.32601427057499])

Regenerating profiles with slit offset option resolved this problem, so I think we can close this ticket.