https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1197 DRP 2-D Pipeline <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=rhl" class="user-hover" rel="rhl">rhl</a> reports:</p> <blockquote> <p>With this how-to-repeat, these plots are pretty-much a smoking gun for a problem with <tt>spectrograph != 1</tt>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> dataId.update(spectrograph=1) spec = butler.get(<span class="code-quote">'pfsArm'</span>, dataId) pfsConfig = butler.get(<span class="code-quote">"pfsConfig"</span>, dataId) ​ from pfs.drp.stella.subtractSky1d <span class="code-keyword">import</span> subtractSky1d sky1d = butler.get(<span class="code-quote">"sky1d"</span>, dataId) subtractSky1d(spec, pfsConfig, sky1d) ​ spec.flux[(spec.mask &amp; ~spec.flags[<span class="code-quote">"REFLINE"</span>]) != 0x0] = np.NaN ​ fig = 1; plt.close(fig); fig = plt.figure(fig) spec.plot(spec.fiberId[100:101], figure=fig, show=False) plt.ylim(-1000, 1000) ​ plt.title(dataId); </pre> </div></div></blockquote> PIPE2D-1197

subtractSky1d applied to spectrograph=3 produces excessive noise

Story Normal Done Done price price Thu, 30 Mar 2023 23:17:38 +0000 Thu, 27 Apr 2023 18:32:45 +0000 Thu, 27 Apr 2023 18:32:45 +0000 0 2 <p>I have a fix that I'm going to put on <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-907" title="Solve profiles for multiple fibers simultaneously" class="issue-link" data-issue-key="PIPE2D-907"><del>PIPE2D-907</del></a> with the other changes for the engineering data release.</p> <p>(It's a completely boring, accounting bug.)</p> <p>This fix will invalidate existing <tt>sky1d</tt> products (they cannot be accessed from the butler any more).</p> <p>The normalisation is still wrong, I think. Here are the calexp, the spectra-subtracted, and the sky-subtracted images for 83149. <span class="image-wrap" style=""><a id="15602_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15602/15602_Figure+1+%2861%29.png" title="Figure 1 (61).png" file-preview-type="image" file-preview-id="15602" file-preview-title="Figure 1 (61).png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15603/_thumb_15603.png" style="border: 0px solid black" /></a></span> <span class="image-wrap" style=""><a id="15601_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15601/15601_Figure+1+%2862%29.png" title="Figure 1 (62).png" file-preview-type="image" file-preview-id="15601" file-preview-title="Figure 1 (62).png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15602/_thumb_15602.png" style="border: 0px solid black" /></a></span> <span class="image-wrap" style=""><a id="15601_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15601/15601_Figure+1+%2862%29.png" title="Figure 1 (62).png" file-preview-type="image" file-preview-id="15601" file-preview-title="Figure 1 (62).png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15601/_thumb_15601.png" style="border: 0px solid black" /></a></span></p> <p>You can also see artefacts from the background subtraction model, but that's a different issue.  The code for the last two plots only differs in doing the sky subtraction:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> pfsArm = butler.get(<span class="code-quote">"pfsArm"</span>, dataId) <span class="code-keyword">if</span> subtractSky: from pfs.drp.stella.subtractSky1d <span class="code-keyword">import</span> subtractSky1d sky1d = butler.get(<span class="code-quote">"sky1d"</span>, dataId) subtractSky1d(pfsArm, pfsConfig, sky1d) </pre> </div></div> <p>Your code doesn't match the figures, so it's not clear to me what you're doing.</p> <p>One thing that I noticed is that when we make an image from spectra, the image includes the values in the <tt>sky</tt> arrays. That's probably not what we want (we only want the <tt>flux</tt>), so I've changed that (on <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-907" title="Solve profiles for multiple fibers simultaneously" class="issue-link" data-issue-key="PIPE2D-907"><del>PIPE2D-907</del></a>).</p> <p>Perhaps that, combined with using the old branch might explain the figures you posted. If not, please provide me detailed instructions on how to reproduce the problem. I've attached my notebook in case it's helpful.</p> <p>The "image includes the values in the <tt>sky</tt> arrays" comment was the clue.  Yes, the code works with the tip of <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-907" title="Solve profiles for multiple fibers simultaneously" class="issue-link" data-issue-key="PIPE2D-907"><del>PIPE2D-907</del></a>; thanks.</p> <p>One thing that would be helpful would be an option to <tt>spectra.makeImage(exp.getDimensions(), traces)</tt> that subtracted the <tt>.sky</tt> array rather than the <tt>.flux</tt>.  Yes;  I can stuff <tt>sky</tt> into <tt>flux</tt>, but it's ugly.</p> <p> </p> <p>This merged as part of <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-907" title="Solve profiles for multiple fibers simultaneously" class="issue-link" data-issue-key="PIPE2D-907"><del>PIPE2D-907</del></a>.</p> <p>I added a toggle to <tt>SpectrumSet.makeImage</tt> to have it construct an image from the <tt>sky</tt> instead of from the <tt>flux</tt>.</p> Development Rank 0|zzswnz: