[#PIPE2D-693] Flux errors near the dichroic are underestimated


RHL asks:
> Have you checked that the quoted errors across the dichroic are correct, in the sense that the red-side and blue-side measurements of the flux at a given wavelength are statistically consistent?


(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-693 $ $PFS_PIPE2D_DIR/weekly/process_weekly.sh -r pipe2d-693 -D /scratch/pprice/pipe2d-693/fixed/

detrend.py /scratch/pprice/pipe2d-693/fixed --calib /scratch/pprice/pipe2d-693/fixed/CALIB --rerun pipe2d-693/pipeline/brn/pipeline --id visit=47 arm=b^r -j 2

import numpy as np
from lsst.daf.persistence import Butler
butler = Butler("/scratch/pprice/pipe2d-693/fixed/rerun/pipe2d-693/pipeline/brn/pipeline")
blueId = dict(visit=47, arm="b")
redId = dict(visit=47, arm="r")
blue = butler.get("pfsArm", blueId)
red = butler.get("pfsArm", redId)
wlMin, wlMax = 643, 645
for ii in range(len(red)):
    redSelect = (red.wavelength[ii] > wlMin) & (red.wavelength[ii] < wlMax)
    redFlux = red.flux[ii][redSelect].mean()
    redErr = np.sqrt(red.variance[ii][redSelect]).mean()
    blueSelect = (blue.wavelength[ii] > wlMin) & (blue.wavelength[ii] < wlMax)
    blueFlux = blue.flux[ii][blueSelect].mean()
    blueErr = np.sqrt(blue.variance[ii][blueSelect]).mean()
    diff = np.abs(redFlux - blueFlux)
    err = np.hypot(redErr, blueErr)
    nSigma = diff/err
    if nSigma > 3:
        print(ii, red.fiberId[ii], blue.fiberId[ii], redFlux, redErr, redSelect.sum(), blueFlux, blueErr, blueSelect.sum())

151 160 160 0.004514443263991073 0.000411609902932711 23 0.04298878209533732 0.0013043212679074675 30
162 171 171 0.12044178672043664 0.0016886640203687363 23 0.13277716863815206 0.00340707000100194 31
163 172 172 0.13493079559900753 0.001793808020948718 23 0.156978592248888 0.0037948187510930113 31
327 364 364 0.1681084379102511 0.0022442062516512913 23 0.13056916431689652 0.0031925625966626276 30
337 374 374 0.12048133641788118 0.001687048880942079 23 0.16057112050003808 0.004201387857562435 31
392 433 433 0.12349895673745459 0.001663794545175998 23 0.18600130173594448 0.004604363788815017 31
396 437 437 0.0038277092440687304 0.0003792050218488275 23 0.1664399803512527 0.04771144933588916 30
419 460 460 0.12122264921626778 0.0016422057235593282 23 0.14605515479435868 0.0036308343087460503 31

index = 392
fiberId = blue.fiberId[index]
import matplotlib.pyplot as plt
plt.plot(blue.wavelength[index], blue.flux[index], "b-")
plt.plot(red.wavelength[index], red.flux[index], "r-")
plt.xlabel("Wavelength (nm)")
plt.ylabel("Flux (uncalibrated)")
plt.show()

There's a large bump in the blue that isn't there in the red.

blueDetMap = butler.get("detectorMap", blueId)
redDetMap = butler.get("detectorMap", redId)
blueImage = butler.get("calexp", blueId)
redImage = butler.get("calexp", redId)

from lsst.afw.display import Display
blueDisplay = Display(backend="ds9", frame=1)
blueDisplay.mtv(blueImage)
blueDetMap.display(blueDisplay, [blue.fiberId[index]], [wlMin, wlMax])
redDisplay = Display(backend="ds9", frame=2)
redDisplay.mtv(redImage)
redDetMap.displindeay(redDisplay, [red.fiberId[index]], [wlMin, wlMax])

It's not visible in the image. It must be in the fiberTrace. It can't be in the flat, since I'm looking at the flattened image.

profiles = butler.get("fiberProfiles", blueId)
prof = profiles[fiberId]
Display(backend="ds9", frame=2).mtv(prof.makeFiberTraceFromDetectorMap(blueDetMap, fiberId).trace)

There's a negative feature that peaks around y=4004, x=-4 relative to the trace --> 647.1 nm.

The bump in the spectrum is peaked at 643.2 nm --> y=3945. There's a smaller negative feature in the trace there too. But that more extreme negative feature doesn't appear to have done anything awful to the spectrum.

Could it be due to some coupling to the neighbouring fibers?
Nothing in their fiberTraces around the row of interest, and no flux stolen from their spectra.

plt.plot(profiles[432].norm, "r-")
plt.plot(profiles[433].norm, "k-")
plt.plot(profiles[434].norm, "b-")
plt.show()

The black line, which is for the troublesome fiber, undercuts the line for both the other fibers, at around the right place. Dividing by this deficit is causing an excess in the extracted spectrum.

plt.plot(profiles[434].norm/profiles[433].norm, "k-")
plt.show()

Yes, there's the bump, peaking at 3945.


constructFiberProfiles.py /scratch/pprice/pipe2d-693/fixed --calib=/scratch/pprice/pipe2d-693/fixed/CALIB --rerun=fiberProfiles --doraise --batch-type=smp --cores=10 --no-versions --clobber-config --id visit=37

butler = Butler("/scratch/pprice/pipe2d-693/fixed/rerun/fiberProfiles")
exposure = butler.get("postISRCCD", visit=37, arm="b")
profiles = FiberProfileSet.readFits("/scratch/pprice/pipe2d-693/fixed/rerun/fiberProfiles/FIBERPROFILES/pfsFiberProfiles-2020-01-01-000037-b1.fits")

plt.plot(profiles[435].norm/profiles[433].norm, "k-")
plt.show()

The bump is gone now. Not sure why. Maybe I got some versions wrong? Let's re-run everything after checking the versions.

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-693 $ $PFS_PIPE2D_DIR/weekly/process_weekly.sh -r pipe2d-693 -D /scratch/pprice/pipe2d-693/revised

The bump is still there.

constructFiberProfiles.py /scratch/pprice/pipe2d-693/revised --calib=/scratch/pprice/pipe2d-693/revised/CALIB --rerun=fiberProfiles --doraise --batch-type=smp --cores=10 --no-versions --clobber-config --id visit=35^37 arm=b
combineFiberProfiles.py pfsFiberProfiles-2020-01-01-000000-b1.fits revised/rerun/fiberProfiles/FIBERPROFILES/pfsFiberProfiles-2020-01-01-00003*.fits

Looks fine. Must be something to do with the timing of the construction of the profiles. Maybe the detectorMap? reduceArc.py is run after constructFiberProfiles.py, so the official profiles must be created using the original detectorMap, which is slightly off. Let's turn on doBlindFind. In fact, it's on by default. But the extraction of the spectra uses the old detectorMap, rather than the results of the blind find! Fix that.

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-693 $ $PFS_PIPE2D_DIR/weekly/process_weekly.sh -r pipe2d-693 -D /scratch/pprice/pipe2d-693/fix2

The bump is gone!

merged = butler.get("pfsMerged", blueId)
merged.plot([433])

Good!

There's something at y=209, at wavelength=389nm, which might be a bit bluer than we care about (<10% throughput). Increased profiles.centerFit.order=9 to get better fits to the traces and ran again as fix3. Ideally, we should fit a detectorMap and measure the profile at the same time (the detectorMap gives the best model of the traces, but the traces are used to extract the spectra, which is (currently) required to do line identification.

Looking for red-blue differences > 3 sigma on this new dataset, we find:

151 160 160 0.004270502681566373 0.0003893999866585177 23 0.03895871326810748 0.0012311061697475381 30

There's a huge spike in the blue, which is due to a CR visible in the image. Ready to call this done.
[PIPE2D-693] Flux errors near the dichroic are underestimated Created: 09/Jan/21 Updated: 20/Feb/21 Resolved: 20/Feb/21
Status:	Done
Project:	DRP 2-D Pipeline
Component/s:	None
Affects Version/s:	None
Fix Version/s:	None
[PIPE2D-693] Flux errors near the dichroic are underestimated Created: 09/Jan/21 Updated: 20/Feb/21 Resolved: 20/Feb/21
