[PIPE2D-968] ReduceExposure 2D sky subtraction fails for SuNSS exposures Created: 24/Jan/22 Updated: 18/Feb/22 Resolved: 18/Feb/22 |
|
| Status: | Done |
| Project: | DRP 2-D Pipeline |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major |
| Reporter: | hassan | Assignee: | hassan |
| Resolution: | Done | Votes: | 0 |
| Labels: | SuNSS | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Story Points: | 3 |
| Sprint: | 2DDRP-2022 A |
| Reviewers: | hassan |
| Description |
|
reduceExposure.py fails with 2D sky subtraction enabled on a representative sky exposure: reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-SuNSS-20211108 --rerun=hassans/sunss/v46472 --doraise --id visit=46472 arm=r -c isr.doFlat=False doSubtractSky2d=True doMeasurePsf=True reduceExposure.photometerLines INFO: Photometered 85817 lines Traceback (most recent call last): File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2022.04/bin/reduceExposure.py", line 25, in <module> ReduceExposureTask.parseAndRun() File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py", line 603, in parseAndRun resultList = taskRunner.run(parsedCmd) File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py", line 221, in run resultList = list(mapFunc(self, targetList)) File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py", line 385, in __call__ result = self.runTask(task, dataRef, kwargs) File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask return task.runDataRef(dataRef, **kwargs) File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2022.04/python/pfs/drp/stella/reduceExposure.py", line 278, in runDataRef linesList, apCorrList) File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2022.04/python/pfs/drp/stella/subtractSky2d.py", line 158, in run image = self.subtractSky(exposure, psf, fiberTrace, detectorMap, pfsConfig, sky2d, apCorr) File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2022.04/python/pfs/drp/stella/subtractSky2d.py", line 288, in subtractSky image = self.makeSkyImage(exposure.getBBox(), psf, fiberTraces, pfsConfig, sky2d, apCorr) File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/drp_stella/w.2022.04/python/pfs/drp/stella/subtractSky2d.py", line 253, in makeSkyImage for wl, flux in zip(model.wavelength, psfFlux): TypeError: zip argument #2 must support iteration |
| Comments |
| Comment by hassan [ 24/Jan/22 ] |
|
I believe the problem is due to how sky fibers are selected in the case of SuNSS exposures. The code assumes a target type of SKY for sky fibers, whereas for SuNSS exposures (specifically the 0xdeadbeef pfsConfig), the target type are either SUNSS_IMAGING or SUNSS_DIFFUSE. So no sky fibers are selected in this instance, and the sky model is essentially None. |
| Comment by price [ 25/Jan/22 ] |
|
Works fine after making the simple fix suggested on Slack: (lsst-scipipe) pprice@tiger2-sumire:~ $ reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-SuNSS-20211108 --rerun=price/pipe2d-968 --doraise --id visit=46472 arm=r -c isr.doFlat=False doSubtractSky2d=True doMeasurePsf=True root INFO: Loading config overrride file '/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/obs_pfs/w.2022.04/config/reduceExposure.py' CameraMapper INFO: Loading exposure registry from /projects/HSC/PFS/Subaru/registry.sqlite3 CameraMapper INFO: Loading calib registry from /projects/HSC/PFS/Subaru/CALIB-SuNSS-20211108/calibRegistry.sqlite3 CameraMapper INFO: Loading calib registry from /projects/HSC/PFS/Subaru/CALIB-SuNSS-20211108/calibRegistry.sqlite3 root INFO: Running: /home/pprice/pfs/drp_stella/bin/reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-SuNSS-20211108 --rerun=price/pipe2d-968 --doraise --id visit=46472 arm=r -c isr.doFlat=False doSubtractSky2d=True doMeasurePsf=True reduceExposure INFO: Processing [{'visit': 46472, 'arm': 'r', 'dateObs': '2021-04-04', 'site': 'S', 'category': 'A', 'spectrograph': 1, 'field': 'SEEING_TEST', 'ccd': 1, 'filter': 'r', 'expTime': 1199.999, 'dataType': 'OBJECT', 'taiObs': '2021-04-04T07:36:36.111', 'pfsDesignId': 3735928559, 'slitOffset': 0.0}] reduceExposure.isr INFO: Performing ISR on sensor {'visit': 46472, 'arm': 'r', 'dateObs': '2021-04-04', 'site': 'S', 'category': 'A', 'spectrograph': 1, 'field': 'SEEING_TEST', 'ccd': 1, 'filter': 'r', 'expTime': 1199.999, 'dataType': 'OBJECT', 'taiObs': '2021-04-04T07:36:36.111', 'pfsDesignId': 3735928559, 'slitOffset': 0.0} reduceExposure.isr INFO: Converting exposure to floating point values reduceExposure.isr INFO: Assembling CCD from amplifiers reduceExposure.isr INFO: Applying bias correction. reduceExposure.isr INFO: Masking defects. reduceExposure.isr INFO: Masking NAN value pixels. reduceExposure.isr INFO: Widening saturation trails. reduceExposure.isr INFO: Applying dark correction. reduceExposure.isr INFO: Interpolating masked pixels. reduceExposure.isr WARN: No rough magnitude zero point set for filter r reduceExposure.isr INFO: Setting rough magnitude zero point: 35.697952 reduceExposure.repair INFO: Identified 2847 cosmic rays. reduceExposure.readLineList INFO: No lamps on; assuming sky. reduceExposure.readLineList INFO: Filtered line lists for lamps: NaI,OH,OI reduceExposure.centroidLines INFO: Measured 85817 line centroids reduceExposure.adjustDetectorMap INFO: Final fit: chi2=118327.581471 dof=49262 xRMS=0.060291 yRMS=0.069859 (0.006032 nm) xSoften=0.031297 ySoften=0.039109 from 24643/30495 lines reduceExposure.adjustDetectorMap INFO: Fit quality from reserved lines: chi2=818844.144452 xRMS=0.062705 yRMS=0.097566 (0.008424 nm) xSoften=0.137860 ySoften=0.473914 from 3388 lines (10.0%) reduceExposure.adjustDetectorMap INFO: Softened fit: chi2=98540.502201 dof=49262 xRMS=0.060655 yRMS=0.071704 (0.006191 nm) xSoften=0.031296 ySoften=0.039105 from 3388 lines reduceExposure.adjustDetectorMap INFO: Softened fit quality from reserved lines: chi2=594514.494636 xRMS=0.062612 yRMS=0.097520 (0.008420 nm) xSoften=0.137848 ySoften=0.473928 from 3388 lines reduceExposure.adjustDetectorMap INFO: Final result: chi2=118340.846722 dof=49262 xRMS=0.060293 yRMS=0.069864 xSoften=0.031297 ySoften=0.039107 from 24643 lines reduceExposure.adjustDetectorMap INFO: Stats for OI: chi2=1955.801216 dof=798 xRMS=0.116545 yRMS=0.074554 xSoften=0.062139 ySoften=0.000000 from 399 lines reduceExposure.adjustDetectorMap INFO: Stats for OH: chi2=116385.045506 dof=48488 xRMS=0.059668 yRMS=0.069831 xSoften=0.030915 ySoften=0.039331 from 24244 lines reduceExposure.adjustDetectorMap INFO: Stats for fiberId=3: chi2=397.851023 dof=94 xRMS=0.133083 yRMS=0.128705 xSoften=0.073909 ySoften=0.076113 from 47 lines (47 OH) reduceExposure.adjustDetectorMap INFO: Stats for fiberId=169: chi2=328.602755 dof=196 xRMS=0.050338 yRMS=0.061800 xSoften=0.017361 ySoften=0.030029 from 98 lines (2 OI, 96 OH) reduceExposure.adjustDetectorMap INFO: Stats for fiberId=306: chi2=502.753543 dof=210 xRMS=0.057093 yRMS=0.070342 xSoften=0.029157 ySoften=0.040794 from 105 lines (2 OI, 103 OH) reduceExposure.adjustDetectorMap INFO: Stats for fiberId=482: chi2=452.784294 dof=202 xRMS=0.051715 yRMS=0.073428 xSoften=0.021869 ySoften=0.041609 from 101 lines (2 OI, 99 OH) reduceExposure.adjustDetectorMap INFO: Stats for fiberId=649: chi2=537.442071 dof=176 xRMS=0.049804 yRMS=0.092030 xSoften=0.021107 ySoften=0.059436 from 88 lines (1 OI, 87 OH) reduceExposure.photometerLines.apertureCorrection.fit INFO: Fit focal plane function: chi^2=226.805675 length=90/90 numSamples=251 numGood=480 numBad=22100 numRejected=10 reduceExposure.photometerLines INFO: Photometered 85817 lines |
| Comment by hassan [ 27/Jan/22 ] |
|
Trivial fix works as stated. Suggest adding a check to see if there are a non-zero number of sky fibers available after this line: https://github.com/Subaru-PFS/drp_stella/blob/32a129274b1f1f20ed86a92cd405bfda66e884fd/python/pfs/drp/stella/subtractSky2d.py#L185 This can be a separate ticket if that's cleaner. |
| Comment by hassan [ 27/Jan/22 ] |
|
Merged to master. |