https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-507 DRP 2-D Pipeline <p>As reported by <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a> in the #commissioning slack channel, the fiber flat construction using recent SM1 data taken at Subaru hangs indefinitely.</p> <p>An example command invocation is as follows:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> constructFiberFlat.py $ROOT --calib $ROOT/CALIB --rerun calib0 --id visit=123..251:3 --batch-type=smp --cores 2 --doraise </pre> </div></div> PIPE2D-507

ConstructFiberFlat.py hangs against Subaru data

Story Major Done Done hassan hassan Fri, 17 Jan 2020 04:13:26 +0000 Wed, 12 Feb 2020 17:29:36 +0000 Wed, 12 Feb 2020 02:44:21 +0000 0 3 <p>The cause of the hanging is the presence of NaN values in the postISRCCD image and variance arrays, that are subsequently passed to the C++ function </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">findCenterPositionsOneTrace</pre> </div></div> <p> and to the </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">MPFitGaussLim</pre> </div></div> <p> fitting routine, which hangs.</p> <p>Replacing the NaN values in the images and variance arrays with non-NaN values seems to fix the problem, but further investigation is required. </p> <p>The cause of the NaN values appearing in the postISRCCD image data also needs to be investigated (most likely as a separate ticket).</p> <p>As a fix is needed in order for the LAM team to process Subaru data effectively, a workaround is being developed.</p> <p>In should be noted that the image mask flags pixels with NaN values correctly, however the <tt>findCenterPositionsOneTrace</tt> code does not use those mask arrays currently.</p> <p>Also note that after a temporary fix to the NaN values describe in the earlier comment, the presence of (close to) zero values in the variance array causes a subsequent exception in the C++ <tt>CurveFitting::fitProfile1d</tt> function. This is also being investigated and a fix is expected soon.</p> <p>Did you run the standard replace-NaNs code, or do something else? When you look at the images, does something jump out at you about which pixels have NaNs? The images can't <img class="emoticon" src="https://pfspipe.ipmu.jp/jira/images/icons/emoticons/help_16.png" height="16" width="16" align="absmiddle" alt="" border="0"/> have NaNs unless there is some problem in the calibration frame; you should check that too</p> <p>I believe this is fixed on <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-391" title="Generate fiberTraces from odd/even flats" class="issue-link" data-issue-key="PIPE2D-391"><del>PIPE2D-391</del></a>, which is ready for merge but blocked by conflicts that I need to sort out. I suggest that in the mean time, we cherry-pick the fix from <a href="https://github.com/Subaru-PFS/obs_pfs/commit/557621242a6ec27d9edd0d9cce1da6df833b3d48" class="external-link" rel="nofollow">obs_pfs</a> and <a href="https://github.com/Subaru-PFS/drp_stella/pull/85/commits/bfd9a2cec18c0796e674d30b73fc5de420d12dff" class="external-link" rel="nofollow">drp_stella</a>.</p> <p>Fix implemented based on changes to drp_stella originally put forward by P Price in <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-391" title="Generate fiberTraces from odd/even flats" class="issue-link" data-issue-key="PIPE2D-391"><del>PIPE2D-391</del></a>, plus variance plane update.<br/> Added additional fix to low variance values, that are not masked. These are set according to a config parameter <tt>minVarRN</tt>, which defines the minimum variance in units of the appropriate amplifier read-out noise. </p> <p>Minor issues reported on the GitHub PR.</p> <p>Fix merged to master. <br/> Separate related problems (eg <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-512" title="ReduceArc.py fails on Subaru data using recent bootstrapped detectormap" class="issue-link" data-issue-key="PIPE2D-512"><del>PIPE2D-512</del></a>) will be addressed in separate tickets.</p> Duplicate PIPE2D-481 Relates PIPE2D-512 Development Rank 0|zzs0lr: Reviewers price Sprint 2DDRP-2020 A Story Points 5.0