https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1310 DRP 2-D Pipeline <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=siegeljc" class="user-hover" rel="siegeljc">Jared Siegel</a> reports a significant degradation in sky subtraction performance following updating from <tt>CALIB-2023-07-v4</tt> to <tt>CALIB-2023-07-v5</tt>. The update only involved fixing the validity ranges for the fiberProfiles, so all arms should now be using the (correct) July fiberProfiles to reduce his exposure except n3 (which didn't have fiberProfiles for April 2023).</p> <p>Jared's reduction script:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> . /scratch/gpfs/HSC/PFS/stack/current/loadLSST.bash setup pfs_pipe2d w.2023.37 reduceExposure.py /projects/HSC/PFS/Subaru --calib=/projects/HSC/PFS/Subaru/CALIB --rerun=siegeljc/10 -j 8 --id visit=97252 -c isr.doFlat=False &lt;Fiber reassignment, to set N% to sky fibers&gt; reduceExposure.py /projects/HSC/PFS/Subaru --calib=/projects/HSC/PFS/Subaru/CALIB --rerun=siegeljc/10 -j 8 --id visit=97252 -c isr.doFlat=False mergeArms.py /projects/HSC/PFS/Subaru --calib=/projects/HSC/PFS/Subaru/CALIB --rerun=siegeljc/10 -j 8 --id visit=97252 </pre> </div></div> PIPE2D-1310

Diagnose sky subtraction problems after updating calibs

Story Normal Done Done price price EDR Thu, 5 Oct 2023 18:22:28 +0000 Fri, 17 Nov 2023 20:37:53 +0000 Fri, 17 Nov 2023 20:37:53 +0000 0 2 <p>Confirmed the changes seen by <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=siegeljc" class="user-hover" rel="siegeljc">Jared Siegel</a>. I have done the sky subtraction on 97252r1 separately for v4 calibs and v5 calibs, and subtracted. There are clear differences, as seen in the plot below.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=arnaud.lefur" class="user-hover" rel="arnaud.lefur">arnaud.lefur</a> has pointed out the existence of "shoulders" in the fiber profiles at +/- 9 pixels with an amplitude of about 1% (both in v4 and v5, but with slight differences). There appears to be a fundamental problem in the construction of the fiber profiles. Given that both v4 and v5 are bad in this regard, I'm going to aim to fix the problem and not worry about the change.</p> <p><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/16545/16545_97252r1_v4-v5.png" width="900" style="border: 0px solid black" /></span></p> <p>I wonder if it comes from flats used for the normalization in the fiberProfiles construction :<br/> 96938..96944 and those were taken without the flat screen.<br/> I would try with 97037..97045 which were taken the day after, with the flat screen in.<br/> Might or might not fix the issue but worth trying.</p> <p>Follow <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1311" title="investigate shoulder features in the fiberProfiles" class="issue-link" data-issue-key="PIPE2D-1311">PIPE2D-1311</a> for the shoulder feature investigation.<br/> In the meantime, I've regenerated fiberProfiles for r1, changing the normalization visits.<br/> <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=siegeljc" class="user-hover" rel="siegeljc">Jared Siegel</a> is currently re-reducing the data using those and hopefully find improved residuals.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=siegeljc" class="user-hover" rel="siegeljc">Jared Siegel</a> confirm that the new r1 profiles did improve greatly the residuals, the results are now closely in line with April data <br/> <span class="image-wrap" style=""><a id="16549_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/16549/16549_97252-newResid_histograms.png" title="97252-newResid_histograms.png" file-preview-type="image" file-preview-id="16549" file-preview-title="97252-newResid_histograms.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/16550/_thumb_16550.png" style="border: 0px solid black" /></a></span> <br/> <span class="image-wrap" style=""><a id="16549_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/16549/16549_97252-newResid_histograms.png" title="97252-newResid_histograms.png" file-preview-type="image" file-preview-id="16549" file-preview-title="97252-newResid_histograms.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/16549/_thumb_16549.png" style="border: 0px solid black" /></a></span> <br/> I'm going forward and now re-building fiberProfiles for the other arm/spectrograph.<br/> I will add those to CALIB-2023-07-v6</p> <p>for m arm, it's slightly trickier because I don't think we've taken any traces with the flat screen in...</p> <p>Turned out the main issue came from the visits used in the fiberProfiles normalization </p> Development Rank 0|zzszl1:y Sprint EDR+AIT