https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-427 DRP 2-D Pipeline <p>I am unable to run <tt>reduceArc.py</tt> on the HgAr data taken at LAM on May 06, 2019.</p> <p>The script that I have used to ingest data and prepare calibrations is at: <tt>/home/ncaplar/ReductionScripts/May21/pfs_preparation_May.sh</tt>. It uses the suggested <tt>pfs_build_calibs.sh</tt>, which can also be found in the same folder.</p> <p>Script that I used, which has <tt>reduceArc.py</tt> command, <tt>pfs_May22_Neon_May_defocus.sh</tt>, is in the same folder.</p> <p>Note that with the same preparation, and using equivalent scripts, Neon and Krypton data passes <tt>reduceArc.py</tt> command. </p> <p>The output from running the final script is at:<br/> <a href="https://gist.github.com/nevencaplar/4131a30fd406f9f4c0d55aea8ccb348a" class="external-link" rel="nofollow">https://gist.github.com/nevencaplar/4131a30fd406f9f4c0d55aea8ccb348a</a></p> PIPE2D-427

reduceArc.py fails with HgAr data taken May 06, 2019

Task Normal Done Done price ncaplar Wed, 22 May 2019 14:05:59 +0000 Thu, 23 May 2019 13:15:55 +0000 Wed, 22 May 2019 22:50:25 +0000 0 3 <p>Excerpt from script output (L485) showing the error is shown below:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> reduceArc FATAL: Failed on dataId={<span class="code-quote">'visit'</span>: 17075, <span class="code-quote">'arm'</span>: <span class="code-quote">'r'</span>, <span class="code-quote">'dateObs'</span>: <span class="code-quote">'2019-05-06'</span>, <span class="code-quote">'site'</span>: <span class="code-quote">'L'</span>, <span class="code-quote">'category'</span>: <span class="code-quote">'A'</span>, <span class="code-quote">'expId'</span>: 17075, <span class="code-quote">'spectrograph'</span>: 1, <span class="code-quote">'field'</span>: <span class="code-quote">'ARC'</span>, <span class="code-quote">'ccd'</span>: 1, <span class="code-quote">'filter'</span>: <span class="code-quote">'r'</span>, <span class="code-quote">'expTime'</span>: 15.0, <span class="code-quote">'dataType'</span>: <span class="code-quote">'arc'</span>, <span class="code-quote">'taiObs'</span>: <span class="code-quote">'2019-05-06'</span>, <span class="code-quote">'pfiDesignId'</span>: 1099528409104, <span class="code-quote">'slitOffset'</span>: 0.0}: Exception: File <span class="code-quote">"src/math/CurveFitting.cc"</span>, line 330, in std::tuple&lt;<span class="code-object">float</span>, ImageT, ImageT, ImageT&gt; pfs::drp::stella::math::{anonymous}::fitProfile1d(<span class="code-keyword">const</span> ndarray::ArrayRef&lt;<span class="code-keyword">const</span> ImageT, 1, 1&gt;&amp;, <span class="code-keyword">const</span> ndarray::ArrayRef&lt;<span class="code-keyword">const</span> ImageT, 1, 1&gt;&amp;, <span class="code-keyword">const</span> ndarray::ArrayRef&lt;bool, 1, 1&gt;&amp;, <span class="code-keyword">const</span> ndarray::ArrayRef&lt;ImageT, 1, 1&gt;&amp;, <span class="code-object">float</span>, bool) [with ImageT = <span class="code-object">float</span>] fitProfile1d:: i = 2: ERROR: dataVar(i) == 0. {0} lsst::pex::exceptions::Exception: <span class="code-quote">'fitProfile1d:: i = 2: ERROR: dataVar(i) == 0.'</span> </pre> </div></div> <p>The error is that there is a negative variance (at 1605,469). It appears to be coming from the dark: the value of the dark is -0.01 per second, and after bias subtraction the pixel is already negative (-20). The variance is set after dark subtraction, so we end up with a negative variance. I'm going to post-process the image after ISR to remove negative variances.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=hassan" class="user-hover" rel="hassan">hassan</a>, could you please review these changes?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> price@MacBook:~/pfs/drp_stella (tickets/PIPE2D-427=) $ git sub commit 5bd04dcf79fadefaada86df2e09577309f7c9145 (HEAD -&gt; tickets/PIPE2D-427, origin/tickets/PIPE2D-427) Author: Paul Price &lt;price@astro.princeton.edu&gt; Date: Wed May 22 15:08:24 2019 -0400 reduceExposure: mask negative variance pixels We have low count levels compared to imaging, so we sometimes have pixels that are negative after bias+dark, which means the variance gets set to a negative number, which causes problems downstream (e.g., CurveFitting.cc:330). Find these pixels, set the variance to infinity and mask them as BAD. python/pfs/drp/stella/reduceExposure.py | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) </pre> </div></div> <p>Reduction passed successfully after these changes. </p> <p>Merged to master.</p> Blocks PIPE2D-414 Relates PIPE2D-430 PIPE2D-428 Development Rank 0|02qpt6:508yjo4 Reviewers hassan Sprint 2DDRP-2019 E Story Points 2.0