[PIPE2D-658] Manual processing of stability test data Created: 01/Dec/20  Updated: 05/Jan/21  Resolved: 08/Dec/20

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Normal
Reporter: hassan Assignee: price
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File opdb_query_stabilitytest.txt     File query_opdb.sh     PNG File stability-20201201.png    
Story Points: 1
Sprint: 2DDRP-2021 A

 Description   

In the absence of a working automated processing environment, please perform a manual processing of recent stability test data taken on:

2020-11-16 (visits 34479..34520)
2020-11-11 (visits 34153..34205)

Also provide an estimate of the accuracy of the wavelength solution (which is typically an output of the reduceArc task).

Later, if additional historical data can be processed, a plot of the wavelength solution accuracy can be made, but that's a future ticket.

To help, the output of a recent query of the opDB for stability test data is attached, along with the query itself.

 Comments   
Comment by price [ 02/Dec/20 ] 
PIPE2D-658: Manual processing of stability test data

This is a new set of stability test data.

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ setup pfs_pipe2d w.2020.48
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ mv /projects/HSC/PFS/Subaru/registry.sqlite3 /projects/HSC/PFS/Subaru/registry.sqlite3.20201130

ingestPfsImages.py /projects/HSC/PFS/Subaru --mode=link --config clobber=True register.ignore=True parse.pfsDesignId=0x0000010001001010 --pfsConfigDir=/projects/HSC/PFS/Subaru/drp/pfsDesign/ -- "/projects/HSC/PFS/Subaru/raw/*/PFSA*" "/projects/HSC/PFS/Subaru/raw/*/sps/PFSA*"

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ mkdir /projects/HSC/PFS/Subaru/CALIB-price/

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ constructPfsBias.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/calibs --id field=BIAS dateObs=2020-11-16 --cores 30
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --validity=1800 --doraise --mode=copy /projects/HSC/PFS/Subaru/rerun/price/pipe2d-658/calibs/BIAS/pfsBias-*.fits
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ constructPfsDark.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/calibs --id field=DARK dateObs=2020-11-16 --cores 30
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --validity=1800 --doraise --mode=copy /projects/HSC/PFS/Subaru/rerun/price/pipe2d-658/calibs/DARK/pfsDark-*.fits

Doesn't look like there's a recent dithered flat. I'll construct one from the previous stability test dataset (it shouldn't have changed much, if any). But to use that, I need biases and darks for that.

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ constructPfsBias.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/calibs --id visit=18218..18227 --cores 30
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --validity=1800 --doraise --mode=copy /projects/HSC/PFS/Subaru/rerun/price/pipe2d-658/calibs/BIAS/pfsBias-*.fits -c clobber=True
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ constructPfsDark.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/calibs --id visit=18188..18202 --cores 30
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --validity=1800 --doraise --mode=copy /projects/HSC/PFS/Subaru/rerun/price/pipe2d-658/calibs/DARK/pfsDark-*.fits -c clobber=True

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --validity=1800 --doraise --mode=copy ~/pfs/drp_pfs_data/detectorMap/detectorMap-sim-*.fits

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ constructFiberFlat.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/calibs --id visit=17839..17982 --cores 30
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --validity=1800 --doraise --mode=copy /projects/HSC/PFS/Subaru/rerun/price/pipe2d-658/calibs/FLAT/pfsFlat-*.fits

Now bootstrap with the first quartz+Neon.

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ bootstrapDetectorMap.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/bootstrap --flatId visit=34481 arm=b --arcId visit=34483 arm=b -c spatialOrder=1

bootstrap INFO: Median difference from detectorMap: 11.249359,-4.296951 pixels
bootstrap INFO: Fit 83/92 points, rms: x=0.707995 y=0.065147 total=0.459968 pixels
bootstrap INFO: Median difference from detectorMap: -0.940176,-6.904669 pixels
bootstrap INFO: Fit 56/62 points, rms: x=0.561787 y=0.041838 total=0.341452 pixels

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ bootstrapDetectorMap.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/bootstrap --flatId visit=34481 arm=r --arcId visit=34483 arm=r --clobber-config

bootstrap INFO: Median difference from detectorMap: -3.967714,-5.123845 pixels
bootstrap INFO: Fit 207/224 points, rms: x=0.698818 y=0.144028 total=0.451050 pixels
bootstrap INFO: Median difference from detectorMap: -3.629265,-3.667544 pixels
bootstrap INFO: Fit 110/151 points, rms: x=0.289558 y=0.065743 total=0.154679 pixels

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --validity=1800 --doraise --mode=copy /projects/HSC/PFS/Subaru/rerun/price/pipe2d-658/bootstrap/DETECTORMAP/pfsDetectorMap-034483-*

constructFiberProfiles.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/calibs --cores 3 --id visit=34488 &
constructFiberProfiles.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/calibs --cores 3 --id visit=34495 &
constructFiberProfiles.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/calibs --cores 3 --id visit=34502 &
constructFiberProfiles.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/calibs --cores 3 --id visit=34509 &
constructFiberProfiles.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/calibs --cores 3 --id visit=34516 &

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ ingestPfsCalibs.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --validity=1800 --doraise --mode=copy /projects/HSC/PFS/Subaru/rerun/price/pipe2d-658/calibs/FIBERPROFILES/pfsFiberProfiles-*.fits


Need PIPE2D-653 fixes to drp_stella for successful detectorMap fit.

Throughput looks pretty rubbish in b1: no lines in any of the arcs except Neon, and even those lines are faint. The quartzes are faint too. Ignore arm=b for now.

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-658 $ reduceArc.py /projects/HSC/PFS/Subaru/ --calib /projects/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-658/arcs --id field=COMPARISON dateObs=2020-11-16 arm=r -j 20

reduceArc.fitDetectorMap INFO: Final fit: chi2=18599.649298 xRMS=0.021147 yRMS=0.023173 (0.001989 nm, 0.742671 km/s) from 8104/10535 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=563520.260577 xRMS=0.043762 yRMS=0.054347 (0.004665 nm, 1.741781 km/s) from 1171 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by 0.014020 pixels (0.001203 nm, 0.449324 km/s) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=16197.397955 xRMS=0.024210 yRMS=0.026618 (0.002285 nm, 0.853097 km/s) from 8104/11706 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=391190.307916 xRMS=0.043247 yRMS=0.054002 (0.004635 nm, 1.730709 km/s) from 1171 lines (10.0%)

HgAr has a line at the top-left corner: not the brightest (~2k peak), but might be helpful to see any shifts.
Neon has multiple bright lines in the bottom-left corner.


HgAr: 34485, 34492, 34499, 34506, 34513, 34520
Neon: 34483, 34490, 34497, 34504, 34511, 34518


import numpy as np
from lsst.daf.persistence import Butler
butler = Butler("/projects/HSC/PFS/Subaru/rerun/price/pipe2d-658/arcs")
visits = np.array([34483, 34490, 34497, 34504, 34511, 34518])
wavelength = 638.4756
fiberId = 650
lines = {vv: butler.get("arcLines", visit=vv, arm="r") for vv in visits}

import itertools
import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 2)
for color, space, (fiberId, wavelength) in zip("krbg", np.linspace(0, 3.0, len(visits)), itertools.product((2, 650), (638.4756, 885.6298))):
    select = [(lines[vv].wavelength == wavelength) & (lines[vv].fiberId == fiberId) for vv in visits]
    xx = np.array([lines[vv].x[ss] for vv, ss in zip(visits, select)]).flatten()
    yy = np.array([lines[vv].y[ss] for vv, ss in zip(visits, select)]).flatten()
    xErr = np.array([lines[vv].xErr[ss] for vv, ss in zip(visits, select)]).flatten()
    yErr = np.array([lines[vv].yErr[ss] for vv, ss in zip(visits, select)]).flatten()
    axes[0].errorbar(visits + space, xx - xx.mean(), yerr=xErr, marker="o", color=color, ecolor=color)
    axes[0].set_xlabel("Visit")
    axes[0].set_ylabel("\\Delta x")
    axes[1].errorbar(visits + space, yy - yy.mean(), yerr=yErr, marker="o", color=color, ecolor=color, label=f"{fiberId}, {wavelength}")
    axes[1].set_xlabel("Visit")
    axes[1].set_ylabel("\\Delta y")
    axes[1].legend()

plt.subplots_adjust(wspace=0.5)
Comment by hassan [ 08/Dec/20 ]

Data processed and corresponding plots were generated.

Generated at Sat Feb 10 15:56:07 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.