[PIPE2D-482]  reduceArc: multiple visits gives poorer result than single visit Created: 28/Nov/19  Updated: 05/Jan/21  Resolved: 27/Mar/20

Status: Won't Fix
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: hassan Assignee: price
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates
relates to PIPE2D-475 Fix coaddition of spectra in ReduceAr... Done
relates to PIPE2D-476 Combine arcs properly Done
Story Points: 1
Sprint: 2DDRP-2019 J, 2DDRP-2021 A
Reviewers: hassan

 Description   

naoki.yasuda reports that despite PIPE2D-475 being fixed, he finds that when running reduceArc.py, single visits at a time leads to a better result than multiple visits. The underlying problem needs to be investigated and fixed.

 Comments   
Comment by hassan [ 28/Nov/19 ]

naoki.yasuda Can you provide the visits you processed to this ticket please?

Comment by naoki.yasuda [ 28/Nov/19 ]

I have used visit=17071..17076 for multiple visits and visit=17075 for a single visit.

Example output for multiple visits is like

reduceArc.calibrateWavelengths INFO: FiberId 525, rms 0.070001 nm (0.817 pix) from 46/47 (0.087720 nm = 1.024 pix for 4 reserved points), 630.94-966.04 nm

and for a single visit

reduceArc.calibrateWavelengths INFO: FiberId 525, rms 0.004030 nm (0.047 pix) from 26/33 (0.004550 nm = 0.053 pix for 4 reserved points), 671.82-966.04 nm

RMS seems to be much smaller for a single visit.

Comment by price [ 18/Mar/20 ]

PIPE2D-476 changed how reduceArc works: it no longer coadds images, but measures lines on different images and then fits them all. I think that has more or less invalidated this ticket. To be sure, I ran on the same data. For the same fiber naoki.yasuda chose, I get for a single visit an RMS of 0.104 pixels, and for multiple visits, 0.077 pixels.

Now, I don't think that this has solved everything. Line identification is sensitive to the choice of matchRadius, which I think is primarily caused by slight errors in the detectorMap: it's been pointed out that we need to revise the bootstrap algorithm to account for the fact that the detector is actually a pair of CCDs. But I think that this exercise has demonstrated that there is now no longer a fundamental discrepancy between single and multiple visits.

pprice@tiger2-sumire:/scratch/pprice/pipe2d-482 $ mkdir DATA
pprice@tiger2-sumire:/scratch/pprice/pipe2d-482 $ echo "lsst.obs.pfs.PfsMapper" > DATA/_mapper
pprice@tiger2-sumire:/scratch/pprice/pipe2d-482 $ makeDummyCableBDesign.py blue green red3 red6
pprice@tiger2-sumire:/scratch/pprice/pipe2d-482 $ makeDummyCableBDesign.py green red1 red4 red6 red8
pprice@tiger2-sumire:/scratch/pprice/pipe2d-482 $ makeDummyCableBDesign.py green red1 red4 red8
pprice@tiger2-sumire:/scratch/pprice/pipe2d-482 $ ingestPfsImages.py DATA '/projects/HSC/PFS/LAM/raw/2019-0[45]-*/PFLA*.fits' --pfsConfigDir .
pprice@tiger2-sumire:/scratch/pprice/pipe2d-482 $ mkdir DATA/CALIB
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-482 $ ingestCalibs.py DATA --calib DATA/CALIB/ ~/pfs/drp_pfs_data/detectorMap/detectorMap-2019Apr-r1.fits --validity 10000 --mode=copy
pprice@tiger2-sumire:/scratch/pprice/pipe2d-482 $ pfs_build_calibs.sh -r calibs -C DATA/CALIB -c 8 -n -b visit=16562..16576 -d visit=16577..16606 -f visit=16612..16740:6 -F visit=16607 DATA

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-482 $ reduceArc.py DATA --calib DATA/CALIB --rerun calibs/arc --id visit=17075 arm=r -j 8 -c identifyLines.matchRadius=0.5

reduceArc.calibrateWavelengths INFO: FiberId 2, rms 0.025042 nm (0.295 pix) from 29/33 (0.028162 nm = 0.332 pix for 10 reserved points), 671.82-966.04 nm
reduceArc.calibrateWavelengths INFO: Fiber 2: wavelength correction -0.004007 +/- 0.081546 nm
reduceArc.calibrateWavelengths INFO: FiberId 63, rms 0.004215 nm (0.049 pix) from 26/34 (0.006594 nm = 0.077 pix for 10 reserved points), 671.82-935.68 nm
reduceArc.calibrateWavelengths INFO: Fiber 63: wavelength correction -0.077705 +/- 0.029938 nm
reduceArc.calibrateWavelengths INFO: FiberId 192, rms 0.003827 nm (0.045 pix) from 25/35 (0.003613 nm = 0.042 pix for 10 reserved points), 671.82-924.56 nm
reduceArc.calibrateWavelengths INFO: Fiber 192: wavelength correction -0.151087 +/- 0.016433 nm
reduceArc.calibrateWavelengths INFO: FiberId 255, rms 0.010412 nm (0.121 pix) from 28/35 (0.013162 nm = 0.153 pix for 10 reserved points), 671.82-949.85 nm
reduceArc.calibrateWavelengths INFO: Fiber 255: wavelength correction -0.151806 +/- 0.007621 nm
reduceArc.calibrateWavelengths INFO: FiberId 339, rms 0.012432 nm (0.144 pix) from 27/36 (0.008252 nm = 0.096 pix for 10 reserved points), 671.82-935.68 nm
reduceArc.calibrateWavelengths INFO: Fiber 339: wavelength correction -0.070652 +/- 0.014877 nm
reduceArc.calibrateWavelengths INFO: FiberId 401, rms 0.010191 nm (0.118 pix) from 27/37 (0.007643 nm = 0.089 pix for 10 reserved points), 671.82-935.68 nm
reduceArc.calibrateWavelengths INFO: Fiber 401: wavelength correction -0.092033 +/- 0.016596 nm
reduceArc.calibrateWavelengths INFO: FiberId 464, rms 0.005364 nm (0.062 pix) from 27/36 (0.005672 nm = 0.066 pix for 10 reserved points), 671.82-935.68 nm
reduceArc.calibrateWavelengths INFO: Fiber 464: wavelength correction -0.115254 +/- 0.027879 nm
reduceArc.calibrateWavelengths INFO: FiberId 525, rms 0.008924 nm (0.104 pix) from 28/37 (0.008733 nm = 0.102 pix for 10 reserved points), 671.82-935.68 nm
reduceArc.calibrateWavelengths INFO: Fiber 525: wavelength correction -0.100386 +/- 0.033164 nm
reduceArc.calibrateWavelengths INFO: FiberId 587, rms 0.002927 nm (0.034 pix) from 21/34 (0.003960 nm = 0.046 pix for 10 reserved points), 671.82-867.03 nm
reduceArc.calibrateWavelengths INFO: Fiber 587: wavelength correction -0.057718 +/- 0.059873 nm
reduceArc.calibrateWavelengths INFO: FiberId 650, rms 0.010758 nm (0.127 pix) from 28/34 (0.027379 nm = 0.322 pix for 10 reserved points), 671.82-935.68 nm
reduceArc.calibrateWavelengths INFO: Fiber 650: wavelength correction -0.012630 +/- 0.042133 nm

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-482 $ reduceArc.py DATA --calib DATA/CALIB --rerun calibs/arc --id visit=17071..17076 arm=r -j 8 -c identifyLines.matchRadius=0.5

reduceArc.calibrateWavelengths INFO: FiberId 2, rms 0.005207 nm (0.061 pix) from 146/188 (0.004989 nm = 0.059 pix for 10 reserved points), 671.82-935.68 nm
reduceArc.calibrateWavelengths INFO: Fiber 2: wavelength correction -0.000654 +/- 0.045689 nm
reduceArc.calibrateWavelengths INFO: FiberId 63, rms 0.009841 nm (0.115 pix) from 162/198 (0.004882 nm = 0.057 pix for 10 reserved points), 637.13-935.68 nm
reduceArc.calibrateWavelengths INFO: Fiber 63: wavelength correction -0.071476 +/- 0.041218 nm
reduceArc.calibrateWavelengths INFO: FiberId 192, rms 0.003841 nm (0.045 pix) from 154/207 (0.002881 nm = 0.034 pix for 10 reserved points), 671.82-935.68 nm
reduceArc.calibrateWavelengths INFO: Fiber 192: wavelength correction -0.151711 +/- 0.015886 nm
reduceArc.calibrateWavelengths INFO: FiberId 255, rms 0.004093 nm (0.048 pix) from 143/217 (0.005197 nm = 0.060 pix for 10 reserved points), 671.82-944.11 nm
reduceArc.calibrateWavelengths INFO: Fiber 255: wavelength correction -0.154494 +/- 0.006561 nm
reduceArc.calibrateWavelengths INFO: FiberId 339, rms 0.007517 nm (0.087 pix) from 165/218 (0.009047 nm = 0.105 pix for 10 reserved points), 671.82-935.68 nm
reduceArc.calibrateWavelengths INFO: Fiber 339: wavelength correction -0.066522 +/- 0.012268 nm
reduceArc.calibrateWavelengths INFO: FiberId 401, rms 0.006458 nm (0.075 pix) from 157/223 (0.003695 nm = 0.043 pix for 10 reserved points), 671.82-935.68 nm
reduceArc.calibrateWavelengths INFO: Fiber 401: wavelength correction -0.090638 +/- 0.019203 nm
reduceArc.calibrateWavelengths INFO: FiberId 464, rms 0.005982 nm (0.070 pix) from 161/228 (0.007797 nm = 0.091 pix for 10 reserved points), 671.82-952.89 nm
reduceArc.calibrateWavelengths INFO: Fiber 464: wavelength correction -0.117851 +/- 0.032010 nm
reduceArc.calibrateWavelengths INFO: FiberId 525, rms 0.006579 nm (0.077 pix) from 162/225 (0.006673 nm = 0.078 pix for 10 reserved points), 671.82-952.89 nm
reduceArc.calibrateWavelengths INFO: Fiber 525: wavelength correction -0.096714 +/- 0.032190 nm
reduceArc.calibrateWavelengths INFO: FiberId 587, rms 0.005941 nm (0.070 pix) from 161/218 (0.004300 nm = 0.050 pix for 10 reserved points), 671.82-935.68 nm
reduceArc.calibrateWavelengths INFO: Fiber 587: wavelength correction -0.064564 +/- 0.041166 nm
reduceArc.calibrateWavelengths INFO: FiberId 650, rms 0.006973 nm (0.082 pix) from 149/214 (0.006440 nm = 0.076 pix for 10 reserved points), 671.82-944.11 nm
reduceArc.calibrateWavelengths INFO: Fiber 650: wavelength correction -0.018627 +/- 0.044291 nm
Comment by price [ 24/Mar/20 ]

I suggest we close this issue without any explicit fix.

Comment by hassan [ 27/Mar/20 ]

I think based on the changes Paul has outlined in his comments, we should close this ticket and create a new one if we find a similar problem.

Generated at Sat Feb 10 15:53:55 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.