https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-725 DRP 2-D Pipeline <p>Running the repairTask on SuNSS data finds almost none of the many CRs in the data. Please fix this, or at least make it less bad.</p> <p> </p> PIPE2D-725

Cosmic ray detection/masking fails for SuNSS data

Story Normal Done Done hassan rhl SuNSS Tue, 16 Feb 2021 23:11:54 +0000 Wed, 17 Mar 2021 03:59:55 +0000 Wed, 17 Mar 2021 03:59:43 +0000 0 3 <p>The main problem appears to be a bad flat field with a large variance set.  However, we'll need to tune params.  I am running with</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> config.interp.modelPsf.defaultFwhm = 3 config.cosmicray.nCrPixelMax = 5000000 </pre> </div></div> <p>(the second line isn't new)</p> <p>We don't have flats for SuNSS data.</p> <p>The problem was that you <em>did</em> make flats for <tt>CALIB-price</tt>, and they had a variance plane set.</p> <p>They were there from a previous experiment. This is an example of why I think it's important everyone is careful about what calibs they use.</p> <p>I agree that people have to be careful, but I think that that having an an official certified CALIB that we can all use is the way to do that. I was not happy to be told, "use CALIB-price" but it's the current workflow.<br/> We can revisit how we do this in a month when the new calibration production tooling is in place.</p> <p>Pushed CR params tuned for SuNSS</p> <p>This may break the simulator when merged to master, and we need to think about how to do this. <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=price" class="user-hover" rel="price">price</a></p> <p>The weekly unfortunately did break following the change to the <tt>obs_pfs</tt> repo. This has been investigated and it turns out to be a problem in fibre identification, where fiber 480 is identified in both visits 35 and 37, and means that the fiberIds for these visits were not actually disjoint prior to being combined. Fiber 480 should not appear in visit 35 in accordance to the <tt>pfsConfig</tt> for that visit, and inspecting the corresponding <tt>calExp</tt>, is clearly a misidentification.</p> <p>This has now been investigated and fixed by <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=price" class="user-hover" rel="price">price</a> in the <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-725" title="Cosmic ray detection/masking fails for SuNSS data" class="issue-link" data-issue-key="PIPE2D-725"><del>PIPE2D-725</del></a> ticket branch of <tt>drp_stella</tt> <a href="https://github.com/Subaru-PFS/drp_stella/commit/b39b96e2520f9c4d9beaab1810e44d58b1e897b5" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/drp_stella/commit/b39b96e2520f9c4d9beaab1810e44d58b1e897b5</a> . </p> <p>The weekly build now completes successfully.</p> <p>Does this mean that you're allowing the slit offsets to float in the weekly?</p> <p>The detectorMap was good enough in this instance. The problem is more to do with finding the traces following a cosmic ray hit. Perhaps <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=price" class="user-hover" rel="price">price</a> can elaborate better.</p> <p>That's the point.  If you tell the code to believe the traces, does it work?  There are options for more general solutions, but we should never need to run them on real data.</p> <p>The problem was not related to the detectorMap. We could change the code to use the detectorMap centers in constructing the profile, which would make that more robust in the steady state, but <tt>constructFiberTrace</tt> runs before <tt>reduceArc</tt>, so we can't always assume we have a perfect detectorMap.</p> <p>I'm confused.  If we use the detectorMap how can we mis-identify fibres?</p> <p>And while I understand the bootstrap problem, in the vast majority of cases we <em>will</em> have a good DetectorMap, the only exception being when we first bring a new detector online (or go from DCB to SuNSS or the PFI), and that's when we need to override trusting the DetectorMap.</p> <p> </p> <p>Currently, the detectorMap is only used for seeding the positions, and trace centroids are coming from the image.</p> <p>And it could be that we were running with blind find.</p> <p>Let's see if turning off blind find solves the problem, it should.  Even blind find shouldn't be running fibre-by-fibre, but that's probably not worth fixing as using it is going to be very unusual.  I have no problem in tweaking the centres, but identifying fibres is something that the detectorMap can do reliably.</p> <p>As mentioned during the internal Princeton group discussion on 2021-03-15: disabling the blind find does result in the weekly passing successfully. </p> <p>As proposed and agreed during that discussion: I will commit the configuration changes in obs_pfs made by RHL and myself, <em>and</em> Paul's changes to drp_stella to make finding fibers more robust when blind find is enabled.</p> <p>Merged changes in <tt>obs_pfs</tt> and {drp_stella}} to master (commits a7647e3 and 34406d3 respectively).</p> Duplicate PIPE2D-728 Development Rank 0|02qpt6:508yl0chzzy000009 Sprint 2DDRP-2021 A 2 2DDRP-2021 A3