https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-482 DRP 2-D Pipeline <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=naoki.yasuda" class="user-hover" rel="naoki.yasuda">naoki.yasuda</a> reports that despite <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-475" title="Fix coaddition of spectra in ReduceArcTask.coaddSpectra" class="issue-link" data-issue-key="PIPE2D-475"><del>PIPE2D-475</del></a> being fixed, he finds that when running <tt>reduceArc.py</tt>, single visits at a time leads to a better result than multiple visits. The underlying problem needs to be investigated and fixed.</p> PIPE2D-482

reduceArc: multiple visits gives poorer result than single visit

Story Normal Won't Fix Won't Fix price hassan Wed, 27 Nov 2019 17:18:38 +0000 Mon, 4 Jan 2021 20:23:10 +0000 Thu, 26 Mar 2020 16:13:38 +0000 0 3 <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=naoki.yasuda" class="user-hover" rel="naoki.yasuda">naoki.yasuda</a> Can you provide the visits you processed to this ticket please?</p> <p>I have used visit=17071..17076 for multiple visits and visit=17075 for a single visit.</p> <p>Example output for multiple visits is like</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> reduceArc.calibrateWavelengths INFO: FiberId 525, rms 0.070001 nm (0.817 pix) from 46/47 (0.087720 nm = 1.024 pix <span class="code-keyword">for</span> 4 reserved points), 630.94-966.04 nm </pre> </div></div> <p>and for a single visit</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> reduceArc.calibrateWavelengths INFO: FiberId 525, rms 0.004030 nm (0.047 pix) from 26/33 (0.004550 nm = 0.053 pix <span class="code-keyword">for</span> 4 reserved points), 671.82-966.04 nm </pre> </div></div> <p>RMS seems to be much smaller for a single visit.</p> <p><a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-476" title="Combine arcs properly" class="issue-link" data-issue-key="PIPE2D-476"><del>PIPE2D-476</del></a> changed how <tt>reduceArc</tt> works: it no longer coadds images, but measures lines on different images and then fits them all. I think that has more or less invalidated this ticket. To be sure, I ran on the same data. For the same fiber <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=naoki.yasuda" class="user-hover" rel="naoki.yasuda">naoki.yasuda</a> chose, I get for a single visit an RMS of 0.104 pixels, and for multiple visits, 0.077 pixels.</p> <p>Now, I don't think that this has solved everything. Line identification is sensitive to the choice of <tt>matchRadius</tt>, which I think is primarily caused by slight errors in the detectorMap: it's been pointed out that we need to revise the bootstrap algorithm to account for the fact that the detector is actually a pair of CCDs. But I think that this exercise has demonstrated that there is now no longer a fundamental discrepancy between single and multiple visits.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> pprice@tiger2-sumire:/scratch/pprice/pipe2d-482 $ mkdir DATA pprice@tiger2-sumire:/scratch/pprice/pipe2d-482 $ echo <span class="code-quote">"lsst.obs.pfs.PfsMapper"</span> &gt; DATA/_mapper pprice@tiger2-sumire:/scratch/pprice/pipe2d-482 $ makeDummyCableBDesign.py blue green red3 red6 pprice@tiger2-sumire:/scratch/pprice/pipe2d-482 $ makeDummyCableBDesign.py green red1 red4 red6 red8 pprice@tiger2-sumire:/scratch/pprice/pipe2d-482 $ makeDummyCableBDesign.py green red1 red4 red8 pprice@tiger2-sumire:/scratch/pprice/pipe2d-482 $ ingestPfsImages.py DATA <span class="code-quote">'/projects/HSC/PFS/LAM/raw/2019-0[45]-*/PFLA*.fits'</span> --pfsConfigDir . pprice@tiger2-sumire:/scratch/pprice/pipe2d-482 $ mkdir DATA/CALIB (lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-482 $ ingestCalibs.py DATA --calib DATA/CALIB/ ~/pfs/drp_pfs_data/detectorMap/detectorMap-2019Apr-r1.fits --validity 10000 --mode=copy pprice@tiger2-sumire:/scratch/pprice/pipe2d-482 $ pfs_build_calibs.sh -r calibs -C DATA/CALIB -c 8 -n -b visit=16562..16576 -d visit=16577..16606 -f visit=16612..16740:6 -F visit=16607 DATA (lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-482 $ reduceArc.py DATA --calib DATA/CALIB --rerun calibs/arc --id visit=17075 arm=r -j 8 -c identifyLines.matchRadius=0.5 reduceArc.calibrateWavelengths INFO: FiberId 2, rms 0.025042 nm (0.295 pix) from 29/33 (0.028162 nm = 0.332 pix <span class="code-keyword">for</span> 10 reserved points), 671.82-966.04 nm reduceArc.calibrateWavelengths INFO: Fiber 2: wavelength correction -0.004007 +/- 0.081546 nm reduceArc.calibrateWavelengths INFO: FiberId 63, rms 0.004215 nm (0.049 pix) from 26/34 (0.006594 nm = 0.077 pix <span class="code-keyword">for</span> 10 reserved points), 671.82-935.68 nm reduceArc.calibrateWavelengths INFO: Fiber 63: wavelength correction -0.077705 +/- 0.029938 nm reduceArc.calibrateWavelengths INFO: FiberId 192, rms 0.003827 nm (0.045 pix) from 25/35 (0.003613 nm = 0.042 pix <span class="code-keyword">for</span> 10 reserved points), 671.82-924.56 nm reduceArc.calibrateWavelengths INFO: Fiber 192: wavelength correction -0.151087 +/- 0.016433 nm reduceArc.calibrateWavelengths INFO: FiberId 255, rms 0.010412 nm (0.121 pix) from 28/35 (0.013162 nm = 0.153 pix <span class="code-keyword">for</span> 10 reserved points), 671.82-949.85 nm reduceArc.calibrateWavelengths INFO: Fiber 255: wavelength correction -0.151806 +/- 0.007621 nm reduceArc.calibrateWavelengths INFO: FiberId 339, rms 0.012432 nm (0.144 pix) from 27/36 (0.008252 nm = 0.096 pix <span class="code-keyword">for</span> 10 reserved points), 671.82-935.68 nm reduceArc.calibrateWavelengths INFO: Fiber 339: wavelength correction -0.070652 +/- 0.014877 nm reduceArc.calibrateWavelengths INFO: FiberId 401, rms 0.010191 nm (0.118 pix) from 27/37 (0.007643 nm = 0.089 pix <span class="code-keyword">for</span> 10 reserved points), 671.82-935.68 nm reduceArc.calibrateWavelengths INFO: Fiber 401: wavelength correction -0.092033 +/- 0.016596 nm reduceArc.calibrateWavelengths INFO: FiberId 464, rms 0.005364 nm (0.062 pix) from 27/36 (0.005672 nm = 0.066 pix <span class="code-keyword">for</span> 10 reserved points), 671.82-935.68 nm reduceArc.calibrateWavelengths INFO: Fiber 464: wavelength correction -0.115254 +/- 0.027879 nm reduceArc.calibrateWavelengths INFO: FiberId 525, rms 0.008924 nm (0.104 pix) from 28/37 (0.008733 nm = 0.102 pix <span class="code-keyword">for</span> 10 reserved points), 671.82-935.68 nm reduceArc.calibrateWavelengths INFO: Fiber 525: wavelength correction -0.100386 +/- 0.033164 nm reduceArc.calibrateWavelengths INFO: FiberId 587, rms 0.002927 nm (0.034 pix) from 21/34 (0.003960 nm = 0.046 pix <span class="code-keyword">for</span> 10 reserved points), 671.82-867.03 nm reduceArc.calibrateWavelengths INFO: Fiber 587: wavelength correction -0.057718 +/- 0.059873 nm reduceArc.calibrateWavelengths INFO: FiberId 650, rms 0.010758 nm (0.127 pix) from 28/34 (0.027379 nm = 0.322 pix <span class="code-keyword">for</span> 10 reserved points), 671.82-935.68 nm reduceArc.calibrateWavelengths INFO: Fiber 650: wavelength correction -0.012630 +/- 0.042133 nm (lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-482 $ reduceArc.py DATA --calib DATA/CALIB --rerun calibs/arc --id visit=17071..17076 arm=r -j 8 -c identifyLines.matchRadius=0.5 reduceArc.calibrateWavelengths INFO: FiberId 2, rms 0.005207 nm (0.061 pix) from 146/188 (0.004989 nm = 0.059 pix <span class="code-keyword">for</span> 10 reserved points), 671.82-935.68 nm reduceArc.calibrateWavelengths INFO: Fiber 2: wavelength correction -0.000654 +/- 0.045689 nm reduceArc.calibrateWavelengths INFO: FiberId 63, rms 0.009841 nm (0.115 pix) from 162/198 (0.004882 nm = 0.057 pix <span class="code-keyword">for</span> 10 reserved points), 637.13-935.68 nm reduceArc.calibrateWavelengths INFO: Fiber 63: wavelength correction -0.071476 +/- 0.041218 nm reduceArc.calibrateWavelengths INFO: FiberId 192, rms 0.003841 nm (0.045 pix) from 154/207 (0.002881 nm = 0.034 pix <span class="code-keyword">for</span> 10 reserved points), 671.82-935.68 nm reduceArc.calibrateWavelengths INFO: Fiber 192: wavelength correction -0.151711 +/- 0.015886 nm reduceArc.calibrateWavelengths INFO: FiberId 255, rms 0.004093 nm (0.048 pix) from 143/217 (0.005197 nm = 0.060 pix <span class="code-keyword">for</span> 10 reserved points), 671.82-944.11 nm reduceArc.calibrateWavelengths INFO: Fiber 255: wavelength correction -0.154494 +/- 0.006561 nm reduceArc.calibrateWavelengths INFO: FiberId 339, rms 0.007517 nm (0.087 pix) from 165/218 (0.009047 nm = 0.105 pix <span class="code-keyword">for</span> 10 reserved points), 671.82-935.68 nm reduceArc.calibrateWavelengths INFO: Fiber 339: wavelength correction -0.066522 +/- 0.012268 nm reduceArc.calibrateWavelengths INFO: FiberId 401, rms 0.006458 nm (0.075 pix) from 157/223 (0.003695 nm = 0.043 pix <span class="code-keyword">for</span> 10 reserved points), 671.82-935.68 nm reduceArc.calibrateWavelengths INFO: Fiber 401: wavelength correction -0.090638 +/- 0.019203 nm reduceArc.calibrateWavelengths INFO: FiberId 464, rms 0.005982 nm (0.070 pix) from 161/228 (0.007797 nm = 0.091 pix <span class="code-keyword">for</span> 10 reserved points), 671.82-952.89 nm reduceArc.calibrateWavelengths INFO: Fiber 464: wavelength correction -0.117851 +/- 0.032010 nm reduceArc.calibrateWavelengths INFO: FiberId 525, rms 0.006579 nm (0.077 pix) from 162/225 (0.006673 nm = 0.078 pix <span class="code-keyword">for</span> 10 reserved points), 671.82-952.89 nm reduceArc.calibrateWavelengths INFO: Fiber 525: wavelength correction -0.096714 +/- 0.032190 nm reduceArc.calibrateWavelengths INFO: FiberId 587, rms 0.005941 nm (0.070 pix) from 161/218 (0.004300 nm = 0.050 pix <span class="code-keyword">for</span> 10 reserved points), 671.82-935.68 nm reduceArc.calibrateWavelengths INFO: Fiber 587: wavelength correction -0.064564 +/- 0.041166 nm reduceArc.calibrateWavelengths INFO: FiberId 650, rms 0.006973 nm (0.082 pix) from 149/214 (0.006440 nm = 0.076 pix <span class="code-keyword">for</span> 10 reserved points), 671.82-944.11 nm reduceArc.calibrateWavelengths INFO: Fiber 650: wavelength correction -0.018627 +/- 0.044291 nm </pre> </div></div> <p>I suggest we close this issue without any explicit fix.</p> <p>I think based on the changes Paul has outlined in his comments, we should close this ticket and create a new one if we find a similar problem.</p> Relates PIPE2D-475 PIPE2D-476 Development Rank 0|02qprw:08 Reviewers hassan Sprint 2DDRP-2019 J 2DDRP-2021 A Story Points 1.0