https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-633 DRP 2-D Pipeline <p><a href="https://pfspipe.ipmu.jp/jira/browse/PIPE1D-14" title="Run 1D-DRP for the resulting products from 2D in the end-to-end processing test" class="issue-link" data-issue-key="PIPE1D-14"><del>PIPE1D-14</del></a> identifies some false lines in coadded spectra from the integration test. They appear to be coming from negative features in the sky spectra. In particular, <tt>visit=25 arm=r</tt> has a single pixel in the fiberTrace at row=116 for the eighth spectrum, which is producing a 3-sigma negative value in the extracted spectrum, but inspection of the <tt>calExp</tt> shows that pixel flux value is well within the expected noise.</p> PIPE2D-633

Debug false lines in extracted spectra

Story Normal Done Done price price Mon, 28 Sep 2020 18:06:43 +0000 Tue, 29 Sep 2020 00:30:16 +0000 Tue, 29 Sep 2020 00:30:16 +0000 0 2 <p>So you're looking at why the 2-D data isn't significant, but the 1-D result is?</p> <p>Is this related to the problem that we use the background as an estimate for its variance? (population v. sample)?</p> <p>That's the first thing I'm investigating. There may be other causes that I will chase down as well.</p> <p>I don't think this is related to how the variance is measured. At the moment, the question is how a single-pixel value of -0.14 with a variance of 21 gets extracted to a spectrum value of -1.0 with a variance of 0.11.</p> <p>Ah, I had mismatched the image and spectrum. The actual image value is -10.8 and variance of 12. That's a S/N of about 3, which matches the S/N in the spectrum. So that question is answered.</p> <p>Now, how is that single 3-sigma low value leaking into everything else?</p> <p>The low value is in a sky fiber, and so the low value is an ingredient for the 1D sky subtraction model, and the subtraction of a low value makes a high value.</p> <p>The 1D sky subtraction is using <tt>FitFocalPlaneTask.fit</tt>, the docstring for which says:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> This implementation is a placeholder, as it simply averages the input vectors instead of doing any real fitting or rejection of outliers, and no attention is paid to the position of the fibers on the focal plane. </pre> </div></div> <p>I should add that there's no treatment of errors either.</p> <p>So I think I now understand what's causing the problem, and the solution is to replace <tt>FitFocalPlaneTask</tt> with a proper implementation. We'll always be limited by the mere two sky fibers in the integration test, but we should be able to make that single bad pixel go away.</p> <p>If we need to add more sky fibres, let's do it.  I'd be OK with a 50:50 ratio for a test suite (as opposed to a guessed 20:80 sky:science on the sky)</p> <p>Debugging is done. Will be fixed on <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-635" title="Improve FitFocalPlaneTask" class="issue-link" data-issue-key="PIPE2D-635"><del>PIPE2D-635</del></a>.</p> Relates PIPE2D-635 PIPE1D-14 Development Rank 0|zzs2rc: