https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-375 DRP 2-D Pipeline <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=ncaplar" class="user-hover" rel="ncaplar">ncaplar</a> reports problems running <tt>reduceArc.py</tt> on recent LAM data:</p> <ul> <li>Raw data in <tt>/tigress/HSC/PFS/LAM/raw/2019-02-02</tt></li> <li>bias=11835..11849, dark=11850..11879, ditheredflats=11535..11663:3, flats=11530..11534, science=11664..11834</li> <li><tt>reduceArc.py $TARGET --calib $TARGET/CALIB --rerun $RERUN/arc --id visit=11748 -j $CORES --config reduceExposure.doSubtractContinuum=True reduceExposure.doWriteCalexp=True</tt></li> <li>The numbers are deduced by looking at <a href="https://people.lam.fr/madec.fabrice/pfs/ait_logbook_SM1.html" class="external-link" rel="nofollow">https://people.lam.fr/madec.fabrice/pfs/ait_logbook_SM1.html</a></li> </ul> <p>It's possible that <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-319" title="Make line identification robust against wavelength shifts" class="issue-link" data-issue-key="PIPE2D-319"><del>PIPE2D-319</del></a> fixes this without any further effort.</p> PIPE2D-375

Investigate reduceArc failures on recent LAM data

Story Normal Done Done price price Thu, 21 Feb 2019 23:10:16 +0000 Thu, 14 Mar 2019 18:36:45 +0000 Thu, 14 Mar 2019 17:01:48 +0000 6.0 0 3 <p>note that science data=11664..11834 is the whole defocused sequence - I am not sure if the code can handle defocused data? Use 11748 for data in the focus (which is the <tt>visit</tt> in the <tt>reduceArc.py</tt>).</p> <p>After a bunch of mucking around with ingesting images on the cluster, I ran the following:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> constructBias.py /tigress/HSC/PFS/LAM --calib /tigress/HSC/PFS/LAM/CALIB --rerun price/pipe2d-375/bias --cores 16 --job bias --id visit=11835..11849 ingestCalibs.py /tigress/HSC/PFS/LAM --calib /tigress/HSC/PFS/LAM/CALIB --validity 3000 /tigress/HSC/PFS/LAM/rerun/price/pipe2d-375/bias/BIAS/pfsBias-2019-02-02-0-r1.fits constructDark.py /tigress/HSC/PFS/LAM --calib /tigress/HSC/PFS/LAM/CALIB --rerun price/pipe2d-375/dark --cores 16 --job dark --id visit=11850..11879 ingestCalibs.py /tigress/HSC/PFS/LAM --calib /tigress/HSC/PFS/LAM/CALIB --validity 3000 /tigress/HSC/PFS/LAM/rerun/price/pipe2d-375/dark/DARK/pfsDark-2019-02-02-0-r1.fits ingestCalibs.py /tigress/HSC/PFS/LAM --calib /tigress/HSC/PFS/LAM/CALIB --validity 3000 /tigress/pprice/drp_stella_data/raw/detectorMap-sim-1-r.fits --mode=copy constructFiberFlat.py /tigress/HSC/PFS/LAM --calib /tigress/HSC/PFS/LAM/CALIB --rerun price/pipe2d-375/flat --cores 16 --job flat --id visit=11535..11663:3 ingestCalibs.py /tigress/HSC/PFS/LAM --calib /tigress/HSC/PFS/LAM/CALIB --validity 3000 /tigress/HSC/PFS/LAM/rerun/price/pipe2d-375/flat/FLAT/pfsFiberFlat-2019-02-01-011535-r1.fits constructFiberTrace.py /tigress/HSC/PFS/LAM --calib /tigress/HSC/PFS/LAM/CALIB --rerun price/pipe2d-375/fiberTrace --cores 1 --job fiberTrace --id visit=11530 ingestCalibs.py /tigress/HSC/PFS/LAM --calib /tigress/HSC/PFS/LAM/CALIB --validity 3000 /tigress/HSC/PFS/LAM/rerun/price/pipe2d-375/fiberTrace/FIBERTRACE/pfsFiberTrace-2019-02-01-011530-r1.fits reduceArc.py /tigress/HSC/PFS/LAM --calib /tigress/HSC/PFS/LAM/CALIB --rerun price/pipe2d-375/arc --id visit=11748 --config reduceExposure.doSubtractContinuum=True reduceExposure.doWriteCalexp=True </pre> </div></div> <p>That last one fails with the following error:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> reduceArc FATAL: Failed on dataId={<span class="code-quote">'visit'</span>: 11748, <span class="code-quote">'dateObs'</span>: <span class="code-quote">'2019-02-02'</span>, <span class="code-quote">'site'</span>: <span class="code-quote">'L'</span>, <span class="code-quote">'category'</span>: <span class="code-quote">'A'</span>, <span class="code-quote">'expId'</span>: 11748, <span class="code-quote">'arm'</span>: <span class="code-quote">'r'</span>, <span class="code-quote">'spectrograph'</span>: 1, <span class="code-quote">'field'</span>: <span class="code-quote">'ARC'</span>, <span class="code-quote">'ccd'</span>: 1, <span class="code-quote">'filter'</span>: <span class="code-quote">'r'</span>, <span class="code-quote">'expTime'</span>: 14.998, <span class="code-quote">'dataType'</span>: <span class="code-quote">'arc'</span>, <span class="code-quote">'taiObs'</span>: <span class="code-quote">'2019-02-02'</span>, <span class="code-quote">'pfiDesignId'</span>: 0, <span class="code-quote">'slitOffset'</span>: 0.0}: AssertionError: Monotonic Traceback (most recent call last): File <span class="code-quote">"/tigress/HSC/PFS/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/16.0+1/python/lsst/pipe/base/cmdLineTask.py"</span>, line 392, in __call__ result = task.run(dataRef, **kwargs) File <span class="code-quote">"/tigress/HSC/PFS/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/drp_stella/5.0/python/pfs/drp/stella/reduceArcTask.py"</span>, line 146, in run results = self.reduceExposure.run([dataRef]) File <span class="code-quote">"/tigress/HSC/PFS/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/drp_stella/5.0/python/pfs/drp/stella/reduceExposure.py"</span>, line 220, in run continua = self.fitContinuum.run(spectra) File <span class="code-quote">"/tigress/HSC/PFS/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/drp_stella/5.0/python/pfs/drp/stella/fitContinuum.py"</span>, line 64, in run continuum = self.wrapArray(self.fitContinuum(spec), spec.fiberId) File <span class="code-quote">"/tigress/HSC/PFS/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/drp_stella/5.0/python/pfs/drp/stella/fitContinuum.py"</span>, line 86, in fitContinuum good = self.maskLines(spectrum) File <span class="code-quote">"/tigress/HSC/PFS/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/drp_stella/5.0/python/pfs/drp/stella/fitContinuum.py"</span>, line 186, in maskLines <span class="code-keyword">assert</span> np.all(delta &gt;= 0) or np.all(delta &lt;= 0), <span class="code-quote">"Monotonic"</span> AssertionError: Monotonic reduceArc FATAL: Failed to process at least one of the components </pre> </div></div> <p>I found a couple of problems.</p> <p>First, this needs an updated detectorMap (I believe the above error is caused by using one I've over-tuned, and is non-monotonic at the very ends). I've generated new detectorMaps from the simulator, and will put them into drp_stella_data as part of the <a href="https://pfspipe.ipmu.jp/jira/browse/SIM2D-94" title="Simulator should accept a PfsConfig instance" class="issue-link" data-issue-key="SIM2D-94"><del>SIM2D-94</del></a>/<a href="https://pfspipe.ipmu.jp/jira/browse/SIM2D-99" title="Build blue-arm data in simulator" class="issue-link" data-issue-key="SIM2D-99"><del>SIM2D-99</del></a>/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-344" title="Implement arm merging" class="issue-link" data-issue-key="PIPE2D-344"><del>PIPE2D-344</del></a>/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-316" title="Demonstrate arm merging" class="issue-link" data-issue-key="PIPE2D-316"><del>PIPE2D-316</del></a> effort.</p> <p>Next, the list of reference lines was incredibly sparse (one or two) because the line list reading code thresholds on the intensity in the NIST list, and the threshold was set over most of the lines. Dropping the threshold, we get lots of lines.</p> <p>Once that was sorted, I tried both the vanilla line identification and the new one on <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-319" title="Make line identification robust against wavelength shifts" class="issue-link" data-issue-key="PIPE2D-319"><del>PIPE2D-319</del></a>. I found that the vanilla version is far superior in terms of the number of lines found and the RMS. I believe this is due to the enormous number of Hg lines, many of which are pretty close to each other; I had previously tested <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-319" title="Make line identification robust against wavelength shifts" class="issue-link" data-issue-key="PIPE2D-319"><del>PIPE2D-319</del></a> with Ne.</p> <p>This works with the changes on <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-316" title="Demonstrate arm merging" class="issue-link" data-issue-key="PIPE2D-316"><del>PIPE2D-316</del></a> (which should be merged soon; note: vanilla line identification algorithm) and the fixed detectorMap if you specify <tt>--config minArcLineIntensity=0</tt>.</p> <p>As discussed on Slack, the Neon data to be tested are:</p> <ul class="alternate" type="square"> <li>Raw data in /tigress/HSC/PFS/LAM/raw/2019-02-16</li> <li>bias=12122..12136, dark=12152..12166, ditheredflats=12172..12300:3, flats=12167..12171, science= 12355..12360</li> <li>The numbers are deduced by looking at <a href="https://people.lam.fr/madec.fabrice/pfs/ait_logbook_SM1.html" class="external-link" rel="nofollow">https://people.lam.fr/madec.fabrice/pfs/ait_logbook_SM1.html</a></li> </ul> <p>This includes a single commit that changes the default reference line intensity threshold to be suitable for HgAr as well as Ne.</p> <p>Merged to master.</p> <p>Fix has been tested against the Neon data Neven mentioned in his comment <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-375?focusedCommentId=15091&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15091" class="external-link" rel="nofollow">https://pfspipe.ipmu.jp/jira/browse/PIPE2D-375?focusedCommentId=15091&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15091</a></p> Relates PIPE2D-379 Development Rank 0|02qpt6:508yjj44 Reviewers hassan Sprint 2019 B 2DDRP-2019 C Story Points 2.0