[PIPE2D-709] Generate recent bootstrapped detectorMap for Subaru Created: 13/Feb/21  Updated: 16/Feb/21  Resolved: 16/Feb/21

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: price Assignee: price
Resolution: Done Votes: 0
Labels: SuNSS
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

To aid reduction of newly-acquired SuNSS data, generate a new bootstrapped detectorMap for Subaru SM1 from Dummy Cable B data just prior to the installation of SuNSS.

 Comments   
Comment by price [ 16/Feb/21 ]

Delivered on Friday.

Found a suitable arc (45744) and flat (45742 for r; 45739 for b).
Biases: 45677..45691
Darks: 45692..45706

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-709 $ generateCommands.py /projects/HSC/PFS/Subaru/ --calib /tigress/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-709 --blocks=pipe2d_709 -j 20 pipe2d-709.yaml pipe2d-709.sh

Need init block (PIPE2D-712).
Bootstrap isn't generated correctly (PIPE2D-713).

Let's run the commands individually ourselves.

constructPfsBias.py /projects/HSC/PFS/Subaru --calib=/tigress/HSC/PFS/Subaru/CALIB-price --rerun=price/pipe2d-709/bias --doraise --batch-type=smp --cores=20 --id visit=45677..45691
ingestPfsCalibs.py /projects/HSC/PFS/Subaru --output=/tigress/HSC/PFS/Subaru/CALIB-price --validity=1800 --doraise --mode=copy -- /projects/HSC/PFS/Subaru/rerun/price/pipe2d-709/bias/BIAS/*.fits
constructPfsDark.py /projects/HSC/PFS/Subaru --calib=/tigress/HSC/PFS/Subaru/CALIB-price --rerun=price/pipe2d-709/dark --doraise --batch-type=smp --cores=20 --id visit=45692..45706
ingestPfsCalibs.py /projects/HSC/PFS/Subaru --output=/tigress/HSC/PFS/Subaru/CALIB-price --validity=1800 --doraise --mode=copy -- /projects/HSC/PFS/Subaru/rerun/price/pipe2d-709/dark/DARK/*.fits

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-709 $ bootstrapDetectorMap.py /projects/HSC/PFS/Subaru --calib=/tigress/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-709/bootstrap --flatId visit=45742 --arcId visit=45744 arm=r -c spatialOrder=1 spectralOrder=1 isr.doFlat=False

bootstrap INFO: Found 452 lines in 10 traces
bootstrap INFO: Matched 376 lines
bootstrap INFO: Median difference from detectorMap: -0.026071,-0.671835 pixels
bootstrap INFO: Fit 214/226 points, rms: x=0.698144 y=0.155658 total=0.452632 pixels
bootstrap INFO: Median difference from detectorMap: -0.145875,-0.682627 pixels
bootstrap INFO: Fit 107/150 points, rms: x=0.285516 y=0.061439 total=0.159123 pixels

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-709 $ bootstrapDetectorMap.py /projects/HSC/PFS/Subaru --calib=/tigress/HSC/PFS/Subaru/CALIB-price --rerun price/pipe2d-709/bootstrap --flatId visit=45739 --arcId visit=45744 arm=b -c spatialOrder=1 spectralOrder=1 isr.doFlat=False

bootstrap INFO: Found 184 lines in 10 traces
bootstrap INFO: Matched 174 lines
bootstrap INFO: Median difference from detectorMap: 0.042010,-0.560648 pixels
bootstrap INFO: Fit 96/104 points, rms: x=0.716973 y=0.073478 total=0.464745 pixels
bootstrap INFO: Median difference from detectorMap: -0.176564,-0.532863 pixels
bootstrap INFO: Fit 64/70 points, rms: x=0.559094 y=0.066189 total=0.324827 pixels

ingestPfsCalibs.py /projects/HSC/PFS/Subaru --output=/tigress/HSC/PFS/Subaru/CALIB-price --validity=1800 --doraise --mode=copy --config clobber=True -- /projects/HSC/PFS/Subaru/rerun/price/pipe2d-709/bootstrap/DETECTORMAP/*.fits

constructFiberProfiles.py /projects/HSC/PFS/Subaru --calib=/tigress/HSC/PFS/Subaru/CALIB-price --rerun=price/pipe2d-709/profiles --doraise --batch-type=smp --cores=20 --id visit=45739^45742 -c isr.doFlat=False

ingestPfsCalibs.py /projects/HSC/PFS/Subaru --output=/tigress/HSC/PFS/Subaru/CALIB-price --validity=1800 --doraise --mode=copy --config clobber=True -- /projects/HSC/PFS/Subaru/rerun/price/pipe2d-709/profiles/FIBERPROFILES/*.fits

reduceArc.py /projects/HSC/PFS/Subaru --calib=/tigress/HSC/PFS/Subaru/CALIB-price --rerun=price/pipe2d-709/detectorMap --doraise --id visit=45743..45746 arm=b -c fitDetectorMap.order=3 reduceExposure.isr.doFlat=False

reduceArc.fitDetectorMap INFO: Final fit: chi2=1035.879258 xRMS=0.095182 yRMS=0.062718 (0.004274 nm, 2.476726 km/s) from 209/418 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=27357.479858 xRMS=0.534432 yRMS=0.195452 (0.013320 nm, 7.718392 km/s) from 47 lines (10.1%)
reduceArc.fitDetectorMap INFO: Softening errors by 0.206139 pixels (0.014048 nm, 8.140385 km/s) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=213.823215 xRMS=0.194306 yRMS=0.228703 (0.015586 nm, 9.031459 km/s) from 209/465 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=773.751027 xRMS=0.360469 yRMS=0.202646 (0.013810 nm, 8.002454 km/s) from 47 lines (10.1%)


Not happy with that result at all, except that I actually got a result.

reduceArc.py /projects/HSC/PFS/Subaru --calib=/tigress/HSC/PFS/Subaru/CALIB-price --rerun=price/pipe2d-709/detectorMap --doraise --id visit=45743..45746 arm=r -c fitDetectorMap.order=3 reduceExposure.isr.doFlat=False

reduceArc.fitDetectorMap INFO: Final fit: chi2=1995.563540 xRMS=0.118448 yRMS=0.086120 (0.007385 nm, 2.757787 km/s) from 269/1315 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=620447.612288 xRMS=0.867036 yRMS=0.324389 (0.027817 nm, 10.387740 km/s) from 146 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by 0.305432 pixels (0.026191 nm, 9.780684 km/s) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=323.601180 xRMS=0.293803 yRMS=0.273889 (0.023486 nm, 8.770592 km/s) from 269/1461 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=1409.068308 xRMS=0.557625 yRMS=0.338038 (0.028987 nm, 10.824803 km/s) from 146 lines (10.0%)

That's even worse!

With PIPE2D-660 (DifferentialDetectorMap), we do MUCH better!
Merged PIPE2D-660.

reduceArc.py /projects/HSC/PFS/Subaru --calib=/tigress/HSC/PFS/Subaru/CALIB-price --rerun=price/pipe2d-709/detectorMap --doraise --id visit=45743..45746 arm=b -c reduceExposure.isr.doFlat=False

reduceArc.fitDetectorMap INFO: Final fit: chi2=673.760298 dof=706 xRMS=0.019120 yRMS=0.025782 (0.001751 nm, 1.014676 km/s) from 381/419 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=113.399519 xRMS=0.084767 yRMS=0.127548 (0.008663 nm, 5.019814 km/s) from 47 lines (10.1%)
reduceArc.fitDetectorMap INFO: Softening errors by 0.008998 pixels (0.000611 nm, 0.354139 km/s) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=705.803279 dof=706 xRMS=0.018530 yRMS=0.024717 (0.001679 nm, 0.972766 km/s) from 381/466 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=120.593685 xRMS=0.085329 yRMS=0.127512 (0.008661 nm, 5.018415 km/s) from 47 lines (10.1%)

reduceArc.py /projects/HSC/PFS/Subaru --calib=/tigress/HSC/PFS/Subaru/CALIB-price --rerun=price/pipe2d-709/detectorMap --doraise --id visit=45743..45746 arm=r -c reduceExposure.isr.doFlat=False

reduceArc.fitDetectorMap INFO: Final fit: chi2=2689.543040 dof=1976 xRMS=0.025168 yRMS=0.026273 (0.002255 nm, 0.841970 km/s) from 1016/1329 lines
reduceArc.fitDetectorMap INFO: Fit quality from reserved lines: chi2=20502.149823 xRMS=0.055518 yRMS=0.048970 (0.004203 nm, 1.569353 km/s) from 148 lines (10.0%)
reduceArc.fitDetectorMap INFO: Softening errors by 0.020690 pixels (0.001776 nm, 0.663039 km/s) to yield chi^2/dof=1
reduceArc.fitDetectorMap INFO: Softened fit: chi2=1967.833799 dof=1976 xRMS=0.033413 yRMS=0.035226 (0.003024 nm, 1.128881 km/s) from 1016/1477 lines
reduceArc.fitDetectorMap INFO: Softened fit quality from reserved lines: chi2=6280.993369 xRMS=0.053251 yRMS=0.049719 (0.004268 nm, 1.593337 km/s) from 148 lines (10.0%)


ingestPfsCalibs.py /projects/HSC/PFS/Subaru --output=/tigress/HSC/PFS/Subaru/CALIB-price --validity=1800 --doraise --mode=copy --config clobber=True -- /projects/HSC/PFS/Subaru/rerun/price/pipe2d-709/detectorMap/DETECTORMAP/*.fits
Generated at Sat Feb 10 15:56:47 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.