https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-968 DRP 2-D Pipeline <p><tt>reduceExposure.py</tt> fails with 2D sky subtraction enabled on a representative sky exposure:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-SuNSS-20211108 --rerun=hassans/sunss/v46472 --doraise --id visit=46472 arm=r -c isr.doFlat=False doSubtractSky2d=True doMeasurePsf=True </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> reduceExposure.photometerLines INFO: Photometered 85817 lines Traceback (most recent call last): File <span class="code-quote">"/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2022.04/bin/reduceExposure.py"</span>, line 25, in &lt;module&gt; ReduceExposureTask.parseAndRun() File <span class="code-quote">"/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py"</span>, line 603, in parseAndRun resultList = taskRunner.run(parsedCmd) File <span class="code-quote">"/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py"</span>, line 221, in run resultList = list(mapFunc(self, targetList)) File <span class="code-quote">"/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py"</span>, line 385, in __call__ result = self.runTask(task, dataRef, kwargs) File <span class="code-quote">"/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py"</span>, line 447, in runTask <span class="code-keyword">return</span> task.runDataRef(dataRef, **kwargs) File <span class="code-quote">"/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2022.04/python/pfs/drp/stella/reduceExposure.py"</span>, line 278, in runDataRef linesList, apCorrList) File <span class="code-quote">"/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2022.04/python/pfs/drp/stella/subtractSky2d.py"</span>, line 158, in run image = self.subtractSky(exposure, psf, fiberTrace, detectorMap, pfsConfig, sky2d, apCorr) File <span class="code-quote">"/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2022.04/python/pfs/drp/stella/subtractSky2d.py"</span>, line 288, in subtractSky image = self.makeSkyImage(exposure.getBBox(), psf, fiberTraces, pfsConfig, sky2d, apCorr) File <span class="code-quote">"/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2022.04/python/pfs/drp/stella/subtractSky2d.py"</span>, line 253, in makeSkyImage <span class="code-keyword">for</span> wl, flux in zip(model.wavelength, psfFlux): TypeError: zip argument #2 must support iteration </pre> </div></div> PIPE2D-968

ReduceExposure 2D sky subtraction fails for SuNSS exposures

Bug Major Done Done hassan hassan SuNSS Sun, 23 Jan 2022 21:16:28 +0000 Thu, 17 Feb 2022 18:27:38 +0000 Thu, 17 Feb 2022 18:27:38 +0000 0 1 <p>I believe the problem is due to how sky fibers are selected in the case of SuNSS exposures.</p> <p><a href="https://github.com/Subaru-PFS/drp_stella/blob/32a129274b1f1f20ed86a92cd405bfda66e884fd/python/pfs/drp/stella/subtractSky2d.py#L185" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/drp_stella/blob/32a129274b1f1f20ed86a92cd405bfda66e884fd/python/pfs/drp/stella/subtractSky2d.py#L185</a></p> <p>The code assumes a target type of <tt>SKY</tt> for sky fibers, whereas for SuNSS exposures (specifically the <tt>0xdeadbeef</tt> pfsConfig), the target type are either SUNSS_IMAGING or SUNSS_DIFFUSE.</p> <p>So no sky fibers are selected in this instance, and the sky model is essentially <tt>None</tt>.</p> <p>Works fine after making the simple fix suggested on Slack:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> (lsst-scipipe) pprice@tiger2-sumire:~ $ reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-SuNSS-20211108 --rerun=price/pipe2d-968 --doraise --id visit=46472 arm=r -c isr.doFlat=False doSubtractSky2d=True doMeasurePsf=True root INFO: Loading config overrride file <span class="code-quote">'/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/obs_pfs/w.2022.04/config/reduceExposure.py'</span> CameraMapper INFO: Loading exposure registry from /projects/HSC/PFS/Subaru/registry.sqlite3 CameraMapper INFO: Loading calib registry from /projects/HSC/PFS/Subaru/CALIB-SuNSS-20211108/calibRegistry.sqlite3 CameraMapper INFO: Loading calib registry from /projects/HSC/PFS/Subaru/CALIB-SuNSS-20211108/calibRegistry.sqlite3 root INFO: Running: /home/pprice/pfs/drp_stella/bin/reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-SuNSS-20211108 --rerun=price/pipe2d-968 --doraise --id visit=46472 arm=r -c isr.doFlat=False doSubtractSky2d=True doMeasurePsf=True reduceExposure INFO: Processing [{<span class="code-quote">'visit'</span>: 46472, <span class="code-quote">'arm'</span>: <span class="code-quote">'r'</span>, <span class="code-quote">'dateObs'</span>: <span class="code-quote">'2021-04-04'</span>, <span class="code-quote">'site'</span>: <span class="code-quote">'S'</span>, <span class="code-quote">'category'</span>: <span class="code-quote">'A'</span>, <span class="code-quote">'spectrograph'</span>: 1, <span class="code-quote">'field'</span>: <span class="code-quote">'SEEING_TEST'</span>, <span class="code-quote">'ccd'</span>: 1, <span class="code-quote">'filter'</span>: <span class="code-quote">'r'</span>, <span class="code-quote">'expTime'</span>: 1199.999, <span class="code-quote">'dataType'</span>: <span class="code-quote">'OBJECT'</span>, <span class="code-quote">'taiObs'</span>: <span class="code-quote">'2021-04-04T07:36:36.111'</span>, <span class="code-quote">'pfsDesignId'</span>: 3735928559, <span class="code-quote">'slitOffset'</span>: 0.0}] reduceExposure.isr INFO: Performing ISR on sensor {<span class="code-quote">'visit'</span>: 46472, <span class="code-quote">'arm'</span>: <span class="code-quote">'r'</span>, <span class="code-quote">'dateObs'</span>: <span class="code-quote">'2021-04-04'</span>, <span class="code-quote">'site'</span>: <span class="code-quote">'S'</span>, <span class="code-quote">'category'</span>: <span class="code-quote">'A'</span>, <span class="code-quote">'spectrograph'</span>: 1, <span class="code-quote">'field'</span>: <span class="code-quote">'SEEING_TEST'</span>, <span class="code-quote">'ccd'</span>: 1, <span class="code-quote">'filter'</span>: <span class="code-quote">'r'</span>, <span class="code-quote">'expTime'</span>: 1199.999, <span class="code-quote">'dataType'</span>: <span class="code-quote">'OBJECT'</span>, <span class="code-quote">'taiObs'</span>: <span class="code-quote">'2021-04-04T07:36:36.111'</span>, <span class="code-quote">'pfsDesignId'</span>: 3735928559, <span class="code-quote">'slitOffset'</span>: 0.0} reduceExposure.isr INFO: Converting exposure to floating point values reduceExposure.isr INFO: Assembling CCD from amplifiers reduceExposure.isr INFO: Applying bias correction. reduceExposure.isr INFO: Masking defects. reduceExposure.isr INFO: Masking NAN value pixels. reduceExposure.isr INFO: Widening saturation trails. reduceExposure.isr INFO: Applying dark correction. reduceExposure.isr INFO: Interpolating masked pixels. reduceExposure.isr WARN: No rough magnitude zero point set <span class="code-keyword">for</span> filter r reduceExposure.isr INFO: Setting rough magnitude zero point: 35.697952 reduceExposure.repair INFO: Identified 2847 cosmic rays. reduceExposure.readLineList INFO: No lamps on; assuming sky. reduceExposure.readLineList INFO: Filtered line lists <span class="code-keyword">for</span> lamps: NaI,OH,OI reduceExposure.centroidLines INFO: Measured 85817 line centroids reduceExposure.adjustDetectorMap INFO: Final fit: chi2=118327.581471 dof=49262 xRMS=0.060291 yRMS=0.069859 (0.006032 nm) xSoften=0.031297 ySoften=0.039109 from 24643/30495 lines reduceExposure.adjustDetectorMap INFO: Fit quality from reserved lines: chi2=818844.144452 xRMS=0.062705 yRMS=0.097566 (0.008424 nm) xSoften=0.137860 ySoften=0.473914 from 3388 lines (10.0%) reduceExposure.adjustDetectorMap INFO: Softened fit: chi2=98540.502201 dof=49262 xRMS=0.060655 yRMS=0.071704 (0.006191 nm) xSoften=0.031296 ySoften=0.039105 from 3388 lines reduceExposure.adjustDetectorMap INFO: Softened fit quality from reserved lines: chi2=594514.494636 xRMS=0.062612 yRMS=0.097520 (0.008420 nm) xSoften=0.137848 ySoften=0.473928 from 3388 lines reduceExposure.adjustDetectorMap INFO: Final result: chi2=118340.846722 dof=49262 xRMS=0.060293 yRMS=0.069864 xSoften=0.031297 ySoften=0.039107 from 24643 lines reduceExposure.adjustDetectorMap INFO: Stats <span class="code-keyword">for</span> OI: chi2=1955.801216 dof=798 xRMS=0.116545 yRMS=0.074554 xSoften=0.062139 ySoften=0.000000 from 399 lines reduceExposure.adjustDetectorMap INFO: Stats <span class="code-keyword">for</span> OH: chi2=116385.045506 dof=48488 xRMS=0.059668 yRMS=0.069831 xSoften=0.030915 ySoften=0.039331 from 24244 lines reduceExposure.adjustDetectorMap INFO: Stats <span class="code-keyword">for</span> fiberId=3: chi2=397.851023 dof=94 xRMS=0.133083 yRMS=0.128705 xSoften=0.073909 ySoften=0.076113 from 47 lines (47 OH) reduceExposure.adjustDetectorMap INFO: Stats <span class="code-keyword">for</span> fiberId=169: chi2=328.602755 dof=196 xRMS=0.050338 yRMS=0.061800 xSoften=0.017361 ySoften=0.030029 from 98 lines (2 OI, 96 OH) reduceExposure.adjustDetectorMap INFO: Stats <span class="code-keyword">for</span> fiberId=306: chi2=502.753543 dof=210 xRMS=0.057093 yRMS=0.070342 xSoften=0.029157 ySoften=0.040794 from 105 lines (2 OI, 103 OH) reduceExposure.adjustDetectorMap INFO: Stats <span class="code-keyword">for</span> fiberId=482: chi2=452.784294 dof=202 xRMS=0.051715 yRMS=0.073428 xSoften=0.021869 ySoften=0.041609 from 101 lines (2 OI, 99 OH) reduceExposure.adjustDetectorMap INFO: Stats <span class="code-keyword">for</span> fiberId=649: chi2=537.442071 dof=176 xRMS=0.049804 yRMS=0.092030 xSoften=0.021107 ySoften=0.059436 from 88 lines (1 OI, 87 OH) reduceExposure.photometerLines.apertureCorrection.fit INFO: Fit focal plane function: chi^2=226.805675 length=90/90 numSamples=251 numGood=480 numBad=22100 numRejected=10 reduceExposure.photometerLines INFO: Photometered 85817 lines </pre> </div></div> <p>Trivial fix works as stated. Suggest adding a check to see if there are a non-zero number of sky fibers available after this line: <a href="https://github.com/Subaru-PFS/drp_stella/blob/32a129274b1f1f20ed86a92cd405bfda66e884fd/python/pfs/drp/stella/subtractSky2d.py#L185" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/drp_stella/blob/32a129274b1f1f20ed86a92cd405bfda66e884fd/python/pfs/drp/stella/subtractSky2d.py#L185</a></p> <p>This can be a separate ticket if that's cleaner.</p> <p>Merged to master.</p> Development Rank 0|zzs9ul:xxjr Reviewers hassan Sprint 2DDRP-2022 A Story Points 3.0