[PIPE2D-1119] No good points in aperture correction Created: 23/Nov/22 Updated: 27/Oct/23 |
|
| Status: | Open |
| Project: | DRP 2-D Pipeline |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Story | Priority: | Normal |
| Reporter: | price | Assignee: | price |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Description |
|
rhl reports an error when running the pipeline on data from the recent engineering run: $ reduceExposure.py /work/drp --calib /work/drp/CALIB --output /work/rhl/rerun/rhl/2022-11 -c doAdjustDetectorMap=True doMeasureLines=True extractSpectra.doCrosstalk=False isr.doFlat=False --id visit=83123 arm=b spectrograph=1 fails with:
reduceExposure FATAL: Failed on dataIds=[{'visit': 83123, 'arm': 'b', 'spectrograph': 1, 'dateObs': '2022-11-21', 'site': 'S', 'category': 'A', 'field': 'M39', 'ccd': 0, 'filter': 'b', 'expTime': 180.005, 'dataType': 'OBJECT', 'taiObs': '2022-11-21T04:29:34.961', 'pfsDesignId': 7362534226687433961, 'slitOffset': 0.0}]: RuntimeError: No good points
Traceback (most recent call last):
File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 436, in __call__
result = self.runTask(task, dataRef, kwargs)
File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 504, in runTask
return task.runDataRef(dataRef, **kwargs)
File "/home/rhl/PFS/drp/stella/python/pfs/drp/stella/reduceExposure.py", line 268, in runDataRef
calibs = self.getSpectralCalibs(sensorRef, exposure, pfsConfig)
File "/home/rhl/PFS/drp/stella/python/pfs/drp/stella/reduceExposure.py", line 543, in getSpectralCalibs
phot = self.photometerLines.run(exposure, lines[lines.description != "Trace"], detectorMap,
File "/home/rhl/PFS/drp/stella/python/pfs/drp/stella/photometerLines.py", line 99, in run
phot = self.photometerLines(exposure, referenceLines, detectorMap, pfsConfig, fiberTraces)
File "/home/rhl/PFS/drp/stella/python/pfs/drp/stella/photometerLines.py", line 176, in photometerLines
apCorr = self.apertureCorrection.run(exposure, pfsConfig, detectorMap, lines)
File "/home/rhl/PFS/drp/stella/python/pfs/drp/stella/apertureCorrections.py", line 125, in run
apCorr = self.fit.run(corrections, pfsConfig, minWavelength=minWavelength,
File "/home/rhl/PFS/drp/stella/python/pfs/drp/stella/fitFocalPlane.py", line 109, in run
raise RuntimeError("No good points")
RuntimeError: No good points
|
| Comments |
| Comment by price [ 24/Nov/22 ] |
|
This is failing in the aperture corrections. There are few lines available to start with in the blue (3 OI and 2 NaI), and for this image most of these are being discarded due to low signal-to-noise (minimum is 10). I see two possible options for dealing with this: I'm leaning towards option #2. While option #1 would allow us to get some sort of measurements, the quality would be questionable. Bad data is worse than no data. The reason we're measuring photometry and aperture corrections is to do 2D sky subtraction, but we don't need to do that here, because there are not a lot of lines, and the signal-to-noise is so poor that 2D sky subtraction is not necessary. |
| Comment by rhl [ 28/Nov/22 ] |
|
I think I'd prefer #1.5. Measure the aperture correction (and its errors, correctly!) and then refuse to apply it to the raw photometry while setting an exit code/quality status. I assume that the aperture corrections are persisted; if not they should be. One of the things that Rubin/PFS pipelines lost from SDSS was the ability to record the quality of a reduction, including flags saying what went wrong. We need this back!
|
| Comment by price [ 29/Nov/22 ] |
|
That sounds ideal, but quality status flags are not something we're in a position to implement for PFS right now. Besides there being no clear solution for this from Rubin (yet), we are still in the in-between Gen2/Gen3 world so we'd have to extend the Rubin solution once it's invented. I'm going with option 2 (which is very close to what you want), and we can modify that once we get a quality status flags implementation from Rubin. Aperture corrections are persisted (butler dataset apCorr). |
| Comment by rhl [ 29/Nov/22 ] |
|
Will you still measure and persist the aperture corrections, but not apply them? |
| Comment by price [ 29/Nov/22 ] |
|
Yes. Although the aperture corrections are NaN. |
| Comment by rhl [ 30/Nov/22 ] |
|
I thought that the idea was to back off the order etc. to do a best-effort; and then not apply them. |
| Comment by price [ 30/Nov/22 ] |
|
If we back off the order, then do we have something that we trust? If we trust it, then we should apply it unreservedly. If we don't, then what was the point of backing off the order? |
| Comment by rhl [ 30/Nov/22 ] |
|
You seemed to not believe the aperture correction, and didn't want to apply it. I'll defer to your judgement here, but there's no downside to calculating it so that we can test its quality. One downside of including a noisy aperture correction is that it'll add noise to every measurement – we need to be propagating the errors in the aperture correction. If it's calculated but not applied it's up to the user to decide what to do. |
| Comment by rhl [ 27/Oct/23 ] |
|
The main problem is fixed on |