https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-972 DRP 2-D Pipeline <p>In the last weekly runs I noticed that some subtraction residuals do not have a properly estimated variance, which makes the 1D pipeline identifying them as real lines. I have already identified four of them at 11020, 11444, 12286, 12350 Angstroms (see attached plots)</p> PIPE2D-972

Unproperly masked/identified sky line residuals

Bug Normal Done Done price vlebrun Fri, 28 Jan 2022 14:09:44 +0000 Mon, 19 Jun 2023 20:01:54 +0000 Wed, 16 Feb 2022 19:48:03 +0000 0 4 <p>a comment after some 10s of  spectra, some of these features appear at the same location in several spectra, but other appear at various location.</p> <p>The <tt>n</tt> arm images have many many sky lines flagged as CRs. In the following image, I've put a green circle (right middle) at the approximate position of a strange feature from the <tt>pfsArm</tt> file. It's a sky line flagged as a CR, as are many other sky lines.<br/> <span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/15092/15092_calexp-1000n.jpg" style="border: 0px solid black" /></span></p> <p>After a configuration tweak, the lines aren't being flagged as CRs so much. Note that this is specific to the simulated data, where the lines are much sharper than in the images from the real instrument.<br/> I've attached some plots of the troublesome lines identified in the original description, showing the problem is much better. The line at 12350 isn't well subtracted, but maybe that's a separate issue.<br/> <span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/15096/15096_fixed_11022.png" style="border: 0px solid black" /></span><br/> <span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/15095/15095_fixed_12286.png" style="border: 0px solid black" /></span><br/> <span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/15094/15094_fixed_12350.png" style="border: 0px solid black" /></span></p> <p>Can you quantify, "lines aren't being flagged as CRs so much"?</p> <p>Also, the original complaint was about sky subtraction residuals. <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=vlebrun" class="user-hover" rel="vlebrun">vlebrun</a>: does that mean that you weren't looking at the CR flags, or are they not being properly propagated?</p> <p>Actually I did not check wether it was CR or sky line residuals (did not even realize there was a flag, we should definitely use it to avoid this kind of trouble), but the fact that I found some at various location seems to indicate that CR might be the explanation</p> <p>You can see in the original spectra posted that there are missing pixels (most conspicuous in the lower green line).</p> <p>As for quantifying, I offer the following from my notes:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Try modifying the FWHM value used by the cosmic ray finder. repair.interp.modelPsf.defaultFwhm=3.0 (<span class="code-keyword">default</span>): reduceExposure.repair INFO: Identified 58346 cosmic rays. repair.interp.modelPsf.defaultFwhm=2.0: reduceExposure.repair INFO: Identified 20666 cosmic rays. repair.interp.modelPsf.defaultFwhm=1.5: reduceExposure.repair INFO: Identified 12285 cosmic rays. repair.interp.modelPsf.defaultFwhm=1.0: reduceExposure.repair INFO: Identified 8376 cosmic rays. Even with defaultFwhm=1.5, getting LOTS of sky lines flagged (though now they tend to be fainter). defaultFwhm=1.0 still gets some sky lines flagged, but looks much better. Can we perhaps <span class="code-keyword">do</span> defaultFwhm=1.5 and modify cond3_fac? Default cond3_fac is 2.5. repair.interp.modelPsf.defaultFwhm=1.5 repair.cosmicray.cond3_fac=10: reduceExposure.repair INFO: Identified 1007 cosmic rays. The result actually looks pretty good. Real CRs are getting flagged, but no (or at least, not many) sky lines. </pre> </div></div> <p>Also dropping this figure posted to Slack:<br/> <span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/15097/15097_calexp-1000n-crFix.png" style="border: 0px solid black" /></span><br/> Right: current defaults. Left: <tt>repair.interp.modelPsf.defaultFwhm=1.5 repair.cosmicray.cond3_fac=10</tt>. The sky lines are left alone, but we’re still finding those real cosmics.</p> <p>Looks good, thanks. We do flag too many pixels around a CR, but that's a different (and not new?) problem.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=vlebrun" class="user-hover" rel="vlebrun">vlebrun</a>, when reporting problems could you also please ensure you record all the necessary information to be able to reproduce the plots? In this case, the <tt>nVisit</tt> and <tt>pfsVisitHash</tt> aren't available, and I had to guess since I have two pfsObjects with different <tt>pfsVisitHash</tt> (I'm guessing because of separate processing in the "weekly" and "science" datasets). Just showing the full filename would be sufficient.</p> <p>And if you can't specify enough information for Paul then we need to fix our book-keeping.</p> <p>Sure I will take care of this (actually we truncate the file name when converting them into local format so it's easy to correct)</p> <p>The info you need should be in the header too</p> <p>The <tt>tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-972" title="Unproperly masked/identified sky line residuals" class="issue-link" data-issue-key="PIPE2D-972"><del>PIPE2D-972</del></a></tt> contains fixes for <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-956" title="Variance is over-estimated in the weekly run" class="issue-link" data-issue-key="PIPE2D-956"><del>PIPE2D-956</del></a> and <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-976" title="Flux of medium resolution arm in simulated pfsObjects looks overestimated" class="issue-link" data-issue-key="PIPE2D-976"><del>PIPE2D-976</del></a> in addition to <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-972" title="Unproperly masked/identified sky line residuals" class="issue-link" data-issue-key="PIPE2D-972"><del>PIPE2D-972</del></a> (all involved improving the quality of science outputs, so I wanted to make sure fixing one didn't break the others).</p> <p>All changes look good.</p> <p>Merged.</p> <p>Just discovered that the changes to pfs_pipe2d (CR parameter tuning for sims data) hadn't been merged, and merged them.</p> Development Rank 0|zzs9ul:xxj4 Reviewers hassan Sprint 2DDRP-2022 A