https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1119 DRP 2-D Pipeline <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=rhl" class="user-hover" rel="rhl">rhl</a> reports an error when running the pipeline on data from the recent engineering run:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ reduceExposure.py /work/drp --calib /work/drp/CALIB --output /work/rhl/rerun/rhl/2022-11 -c doAdjustDetectorMap=True doMeasureLines=True extractSpectra.doCrosstalk=False isr.doFlat=False --id visit=83123 arm=b spectrograph=1 </pre> </div></div> <p>fails with:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> reduceExposure FATAL: Failed on dataIds=[{<span class="code-quote">'visit'</span>: 83123, <span class="code-quote">'arm'</span>: <span class="code-quote">'b'</span>, <span class="code-quote">'spectrograph'</span>: 1, <span class="code-quote">'dateObs'</span>: <span class="code-quote">'2022-11-21'</span>, <span class="code-quote">'site'</span>: <span class="code-quote">'S'</span>, <span class="code-quote">'category'</span>: <span class="code-quote">'A'</span>, <span class="code-quote">'field'</span>: <span class="code-quote">'M39'</span>, <span class="code-quote">'ccd'</span>: 0, <span class="code-quote">'filter'</span>: <span class="code-quote">'b'</span>, <span class="code-quote">'expTime'</span>: 180.005, <span class="code-quote">'dataType'</span>: <span class="code-quote">'OBJECT'</span>, <span class="code-quote">'taiObs'</span>: <span class="code-quote">'2022-11-21T04:29:34.961'</span>, <span class="code-quote">'pfsDesignId'</span>: 7362534226687433961, <span class="code-quote">'slitOffset'</span>: 0.0}]: RuntimeError: No good points Traceback (most recent call last): File <span class="code-quote">"/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py"</span>, line 436, in __call__ result = self.runTask(task, dataRef, kwargs) File <span class="code-quote">"/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py"</span>, line 504, in runTask <span class="code-keyword">return</span> task.runDataRef(dataRef, **kwargs) File <span class="code-quote">"/home/rhl/PFS/drp/stella/python/pfs/drp/stella/reduceExposure.py"</span>, line 268, in runDataRef calibs = self.getSpectralCalibs(sensorRef, exposure, pfsConfig) File <span class="code-quote">"/home/rhl/PFS/drp/stella/python/pfs/drp/stella/reduceExposure.py"</span>, line 543, in getSpectralCalibs phot = self.photometerLines.run(exposure, lines[lines.description != <span class="code-quote">"Trace"</span>], detectorMap, File <span class="code-quote">"/home/rhl/PFS/drp/stella/python/pfs/drp/stella/photometerLines.py"</span>, line 99, in run phot = self.photometerLines(exposure, referenceLines, detectorMap, pfsConfig, fiberTraces) File <span class="code-quote">"/home/rhl/PFS/drp/stella/python/pfs/drp/stella/photometerLines.py"</span>, line 176, in photometerLines apCorr = self.apertureCorrection.run(exposure, pfsConfig, detectorMap, lines) File <span class="code-quote">"/home/rhl/PFS/drp/stella/python/pfs/drp/stella/apertureCorrections.py"</span>, line 125, in run apCorr = self.fit.run(corrections, pfsConfig, minWavelength=minWavelength, File <span class="code-quote">"/home/rhl/PFS/drp/stella/python/pfs/drp/stella/fitFocalPlane.py"</span>, line 109, in run raise RuntimeError(<span class="code-quote">"No good points"</span>) RuntimeError: No good points </pre> </div></div> PIPE2D-1119

No good points in aperture correction

Story Normal Open Unresolved price price Wed, 23 Nov 2022 01:56:06 +0000 Thu, 26 Oct 2023 21:11:02 +0000 0 2 <p>This is failing in the aperture corrections. There are few lines available to start with in the blue (3 OI and 2 NaI), and for this image most of these are being discarded due to low signal-to-noise (minimum is 10). I see two possible options for dealing with this:<br/> 1. Attempt to apply an aperture correction by backing off the polynomial order and/or the signal-to-noise limit.<br/> 2. Give up on measuring line photometry in this case.</p> <p>I'm leaning towards option #2. While option #1 would allow us to get some sort of measurements, the quality would be questionable. Bad data is worse than no data. The reason we're measuring photometry and aperture corrections is to do 2D sky subtraction, but we don't need to do that here, because there are not a lot of lines, and the signal-to-noise is so poor that 2D sky subtraction is not necessary.</p> <p>I think I'd prefer #1.5.   Measure the aperture correction (and its errors, correctly!) and then refuse to apply it to the raw photometry while setting an exit code/quality status.  I assume that the aperture corrections are persisted; if not they should be.</p> <p>One of the things that Rubin/PFS pipelines lost from SDSS was the ability to record the quality of a reduction, including flags saying what went wrong.  We need this back!</p> <p> </p> <p>That sounds ideal, but quality status flags are not something we're in a position to implement for PFS right now. Besides there being no clear solution for this from Rubin (yet), we are still in the in-between Gen2/Gen3 world so we'd have to extend the Rubin solution once it's invented. I'm going with option 2 (which is very close to what you want), and we can modify that once we get a quality status flags implementation from Rubin.</p> <p>Aperture corrections are persisted (butler dataset <tt>apCorr</tt>).</p> <p>Will you still measure and persist the aperture corrections, but not apply them?</p> <p>Yes. Although the aperture corrections are <tt>NaN</tt>.</p> <p>I thought that the idea was to back off the order etc. to do a best-effort; and then not apply them.  </p> <p>If we back off the order, then do we have something that we trust? If we trust it, then we should apply it unreservedly. If we don't, then what was the point of backing off the order?</p> <p>You seemed to not believe the aperture correction, and didn't want to apply it.   I'll defer to your judgement here, but there's no downside to calculating it so that we can test its quality.</p> <p>One downside of including a noisy aperture correction is that it'll add noise to every measurement – we need to be propagating the errors in the aperture correction.  If it's calculated but not applied it's up to the user to decide what to do.</p> <p>The main problem is fixed on <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1322" title="Inability to fit aperture corrections can lead to reduceExposure.py failing" class="issue-link" data-issue-key="PIPE2D-1322"><del>PIPE2D-1322</del></a>, setting the fluxes and flux errors to <tt>NaN</tt>. I'm leaving this ticket open as a reminder that this is not really the best solution (as discussed here), but it is no longer urgent.</p> Relates PIPE2D-1322 Development Rank 0|02qpjd:00r20060i22