https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1063 DRP 2-D Pipeline <p>The simulator produces sharper features than the actual instrument does, so fiber traces often get mis-identified as CRs. This is especially bad in the NIR, where there are lots of bright sky lines, and when working with bright targets.</p> <p>Tune the CR parameters for the integration test and the weekly+science to minimise CR false positives.</p> PIPE2D-1063

Tune CR parameters for simulator data

Story Normal Done Done price price Thu, 21 Jul 2022 18:55:28 +0000 Fri, 14 Oct 2022 06:12:33 +0000 Tue, 11 Oct 2022 21:43:53 +0000 0 6 <p>Wouldn't it be easier and more realistic to degrade the spots a little?</p> <p>Discovered that the simulator and Subaru PSFs aren't extremely different (x and y sigma for 2D Gaussian fit to arc lines was about 1.3 for both), and that we're having the same problems with CR finding on the Subaru quartz observations as we're having with the simulator. Tried tuning CR parameters but wasn't able to reliably find CRs while avoiding false positives from the traces. Decided to try subtracting a vertical median filter to remove the traces before CR finding, and this works very nicely, finding CRs that I had trouble identifying visually on the original. It does find a sprinkling of false-positives, but they are not systematically distributed along the traces as they were before. Perhaps we might benefit from careful parameter tuning, but this change already results in a dramatic improvement in CR finding quality.</p> <p>Can you attach some images?  This sounds like a good and simple approach, but something more sophisticated (extraction; median filter; project back onto traces; subtract) might work better.</p> <p>The below image shows the <tt>calexp</tt> image, the trace-subtracted image, and the <tt>calexp</tt> mask.</p> <p>I tried the more sophisticated approach, but currently extraction and image generation from fiber traces introduces step functions into the data that makes CR finding much worse; I'm hoping <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-907" title="Solve profiles for multiple fibers simultaneously" class="issue-link" data-issue-key="PIPE2D-907"><del>PIPE2D-907</del></a> (and odd-even quartz observations) will fix this.</p> <p><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/15316/15316_pipe2d-1063.jpg" style="border: 0px solid black" /></span></p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=kiyoto.yabe" class="user-hover" rel="kiyoto.yabe">Kiyoto Yabe</a> ran the weekly with this change and found poor redshift performance, but it was not clear that CR masking was to blame.</p> <p>Note: I need to convert the Gen3 pipeline to also implement the trace subtraction before searching for CRs. This will involve moving things around some, as I want to do it after adjusting the detectorMap (so we have an opportunity to mask sky lines).</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=kiyoto.yabe" class="user-hover" rel="kiyoto.yabe">Kiyoto Yabe</a> writes:</p> <blockquote> <p>The 1D redshift determination is much improved with the new one (although there are some mis-identification but only a few). The CRs (and other features) are masked properly in this version as far as I checked.</p></blockquote> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=rhl" class="user-hover" rel="rhl">rhl</a> requests checking <tt>visit=80748 arm=r</tt>, where sky lines are being masked as CRs.</p> <p>Code changes provided in pull requests look fine to me. Minor comments in PR for <tt>drp_stella</tt>: <a href="https://github.com/Subaru-PFS/drp_stella/pull/284" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/drp_stella/pull/284</a></p> <p>Ran <tt>visit=80748 arm=r</tt> with the ticket branch. CRs are getting masked just fine, and sky lines are not causing false positives.</p> <p>Merged.</p> <p>I confirm that things are going back to normal, 93% success rate in  the redshift measurement in the latest weekly run </p> <p>But this merged <b>after</b> the most recent weekly...?</p> <p>ah... the improvements come from other modification then? but the fact is that the bad pixels I used to see in the September weekly runs are not present in the October 10 one...</p> <p>Maybe the data you received were processed with this branch, but it could not have been part of the Proper weekly.</p> <p>According to header, drp_stella version is w.2022.40a-12-g033bd60+1</p> <p>OK, so that's this branch on top of the last weekly.</p> <p>Yes, I forgot to switch back to the <em>normal</em> mode after I checked the new fix, so yes, the results are from that branch. Sorry.</p> <p>well, don't be sorry, now we know that the fix worked fine<img class="emoticon" src="https://pfspipe.ipmu.jp/jira/images/icons/emoticons/biggrin.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> Relates PIPE2D-188 PIPE1D-63 Development Rank 0|02qpjd:00r202o Reviewers hassan Sprint 2DDRP-2022 F Story Points 2.0