[PIPE2D-841] constructPfsDark can silently generate an all-NaN master dark Created: 15/May/21  Updated: 18/Jun/21  Resolved: 18/Jun/21

Status: Won't Fix
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: rhl Assignee: price
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: JPEG File dark.jpg    
Story Points: 1
Sprint: 2DDRP-2021 A5, 2DDRP-2021 A 6

 Description   

I was looking at a curious defect that's present in e.g. 11150 but not in the darks created using 12137.. 12166, so ran the command

constructPfsDark.py /projects/HSC/PFS/LAM --rerun rhl/dark --id visit=11510..11514 -j 10 --batch-type none  -c isr.doBias=False

(there are no biases registered for that data, but that's a different problem). When I went to look at the resulting master bias it turned out to be all NaN with NO_DATA set.

Looking back at the logs there are signs of trouble:

dark INFO: Scatter combination
dark INFO: Combining {'arm': 'r', 'spectrograph': 1, 'ccd': 1, 'filter': 'NONE', 'calibTime': '2019-02-01T19:56:44.999998720', 'visit0': 11510, 'calibDate': '2019-02-01'} on tiger2-sumire.princeton.edu:212211
/tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3118: RuntimeWarning: Mean of empty slice.
  out=out, **kwargs)
/tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/core/_methods.py:85: RuntimeWarning: invalid value encountered in true_divide
  ret = ret.dtype.type(ret / rcount)
dark INFO: Writing {'arm': 'r', 'spectrograph': 1, 'ccd': 1, 'filter': 'NONE', 'calibTime': '2019-02-01T19:56:44.999998720', 'visit0': 11510, 'calibDate': '2019-02-01'} on tiger2-sumire.princeton.edu:212211

All the visits look fine by eye but even if I did something stupid we need to improve the diagnostics. If this is all going to be replaced with new code from LSST that'd be OK too, but the fact that we have PFS-flavoured versions of these routines makes me a little concerned that that won't be a panacea.

 Comments   
Comment by price [ 18/Jun/21 ]

I cannot reproduce this. There are no signs of trouble in the log, and the output image looks fine.

(lsst-scipipe) pprice@tiger2-sumire:~/pfs/drp_stella[master%] $ constructPfsDark.py /projects/HSC/PFS/LAM --rerun price/pipe2d-841 --id visit=11510..11514 --batch-type=none -j 10  -c isr.doBias=False
root INFO: Loading config overrride file '/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/obs_pfs/w.2021.23/config/dark.py'
CameraMapper INFO: Loading exposure registry from /projects/HSC/PFS/LAM/registry.sqlite3
CameraMapper INFO: Loading calib registry from /projects/HSC/PFS/LAM/CALIB/calibRegistry.sqlite3
root INFO: Loading config overrride file '/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/obs_pfs/w.2021.23/config/dark.py'
CameraMapper INFO: Loading exposure registry from /projects/HSC/PFS/LAM/registry.sqlite3
CameraMapper INFO: Loading calib registry from /projects/HSC/PFS/LAM/CALIB/calibRegistry.sqlite3
root INFO: Running: /home/pprice/pfs/drp_stella/bin/constructPfsDark.py /projects/HSC/PFS/LAM --rerun price/pipe2d-841 --id visit=11510..11514 -j 10 -c isr.doBias=False
CameraMapper INFO: Loading exposure registry from /projects/HSC/PFS/LAM/registry.sqlite3
CameraMapper INFO: Loading calib registry from /projects/HSC/PFS/LAM/CALIB/calibRegistry.sqlite3
CameraMapper INFO: Loading exposure registry from /projects/HSC/PFS/LAM/registry.sqlite3
CameraMapper INFO: Loading calib registry from /projects/HSC/PFS/LAM/CALIB/calibRegistry.sqlite3
dark INFO: Scatter processing
dark INFO: Processing DataId(initialdata={'visit': 11510, 'dateObs': '2019-02-01', 'site': 'L', 'category': 'A', 'arm': 'r', 'spectrograph': 1, 'field': 'DARK', 'ccd': 1, 'filter': 'r', 'expTime': 300.0, 'dataType': 'DARK', 'taiObs': '2019-02-01T19:44:08', 'pfsDesignId': 1099528409104, 'slitOffset': 0.0}, tag=set()) on tiger2-sumire.princeton.edu:189253
dark.isr INFO: Performing ISR on sensor DataId(initialdata={'visit': 11510, 'dateObs': '2019-02-01', 'site': 'L', 'category': 'A', 'arm': 'r', 'spectrograph': 1, 'field': 'DARK', 'ccd': 1, 'filter': 'r', 'expTime': 300.0, 'dataType': 'DARK', 'taiObs': '2019-02-01T19:44:08', 'pfsDesignId': 1099528409104, 'slitOffset': 0.0}, tag=set())
dark.isr INFO: Converting exposure to floating point values
dark.isr INFO: Assembling CCD from amplifiers
dark.isr INFO: Masking defects.
dark.isr INFO: Masking NAN value pixels.
dark.isr INFO: Widening saturation trails.
dark.isr INFO: Interpolating masked pixels.
dark.isr WARN: No rough magnitude zero point set for filter r
dark.isr INFO: Setting rough magnitude zero point: 34.192803
dark.repair INFO: Identified 341 cosmic rays.
dark INFO: Processing DataId(initialdata={'visit': 11511, 'dateObs': '2019-02-01', 'site': 'L', 'category': 'A', 'arm': 'r', 'spectrograph': 1, 'field': 'DARK', 'ccd': 1, 'filter': 'r', 'expTime': 300.0, 'dataType': 'DARK', 'taiObs': '2019-02-01T19:50:08', 'pfsDesignId': 1099528409104, 'slitOffset': 0.0}, tag=set()) on tiger2-sumire.princeton.edu:189253
dark.isr INFO: Performing ISR on sensor DataId(initialdata={'visit': 11511, 'dateObs': '2019-02-01', 'site': 'L', 'category': 'A', 'arm': 'r', 'spectrograph': 1, 'field': 'DARK', 'ccd': 1, 'filter': 'r', 'expTime': 300.0, 'dataType': 'DARK', 'taiObs': '2019-02-01T19:50:08', 'pfsDesignId': 1099528409104, 'slitOffset': 0.0}, tag=set())
dark.isr INFO: Converting exposure to floating point values
dark.isr INFO: Assembling CCD from amplifiers
dark.isr INFO: Masking defects.
dark.isr INFO: Masking NAN value pixels.
dark.isr INFO: Widening saturation trails.
dark.isr INFO: Interpolating masked pixels.
dark.isr WARN: No rough magnitude zero point set for filter r
dark.isr INFO: Setting rough magnitude zero point: 34.192803
dark.repair INFO: Identified 399 cosmic rays.
dark INFO: Processing DataId(initialdata={'visit': 11512, 'dateObs': '2019-02-01', 'site': 'L', 'category': 'A', 'arm': 'r', 'spectrograph': 1, 'field': 'DARK', 'ccd': 1, 'filter': 'r', 'expTime': 300.0, 'dataType': 'DARK', 'taiObs': '2019-02-01T19:56:08', 'pfsDesignId': 1099528409104, 'slitOffset': 0.0}, tag=set()) on tiger2-sumire.princeton.edu:189253
dark.isr INFO: Performing ISR on sensor DataId(initialdata={'visit': 11512, 'dateObs': '2019-02-01', 'site': 'L', 'category': 'A', 'arm': 'r', 'spectrograph': 1, 'field': 'DARK', 'ccd': 1, 'filter': 'r', 'expTime': 300.0, 'dataType': 'DARK', 'taiObs': '2019-02-01T19:56:08', 'pfsDesignId': 1099528409104, 'slitOffset': 0.0}, tag=set())
dark.isr INFO: Converting exposure to floating point values
dark.isr INFO: Assembling CCD from amplifiers
dark.isr INFO: Masking defects.
dark.isr INFO: Masking NAN value pixels.
dark.isr INFO: Widening saturation trails.
dark.isr INFO: Interpolating masked pixels.
dark.isr WARN: No rough magnitude zero point set for filter r
dark.isr INFO: Setting rough magnitude zero point: 34.192803
dark.repair INFO: Identified 368 cosmic rays.
dark INFO: Processing DataId(initialdata={'visit': 11513, 'dateObs': '2019-02-01', 'site': 'L', 'category': 'A', 'arm': 'r', 'spectrograph': 1, 'field': 'DARK', 'ccd': 1, 'filter': 'r', 'expTime': 300.0, 'dataType': 'DARK', 'taiObs': '2019-02-01T20:02:08', 'pfsDesignId': 1099528409104, 'slitOffset': 0.0}, tag=set()) on tiger2-sumire.princeton.edu:189253
dark.isr INFO: Performing ISR on sensor DataId(initialdata={'visit': 11513, 'dateObs': '2019-02-01', 'site': 'L', 'category': 'A', 'arm': 'r', 'spectrograph': 1, 'field': 'DARK', 'ccd': 1, 'filter': 'r', 'expTime': 300.0, 'dataType': 'DARK', 'taiObs': '2019-02-01T20:02:08', 'pfsDesignId': 1099528409104, 'slitOffset': 0.0}, tag=set())
dark.isr INFO: Converting exposure to floating point values
dark.isr INFO: Assembling CCD from amplifiers
dark.isr INFO: Masking defects.
dark.isr INFO: Masking NAN value pixels.
dark.isr INFO: Widening saturation trails.
dark.isr INFO: Interpolating masked pixels.
dark.isr WARN: No rough magnitude zero point set for filter r
dark.isr INFO: Setting rough magnitude zero point: 34.192803
dark.repair INFO: Identified 363 cosmic rays.
dark INFO: Processing DataId(initialdata={'visit': 11514, 'dateObs': '2019-02-01', 'site': 'L', 'category': 'A', 'arm': 'r', 'spectrograph': 1, 'field': 'DARK', 'ccd': 1, 'filter': 'r', 'expTime': 300.0, 'dataType': 'DARK', 'taiObs': '2019-02-01T20:08:08', 'pfsDesignId': 1099528409104, 'slitOffset': 0.0}, tag=set()) on tiger2-sumire.princeton.edu:189253
dark.isr INFO: Performing ISR on sensor DataId(initialdata={'visit': 11514, 'dateObs': '2019-02-01', 'site': 'L', 'category': 'A', 'arm': 'r', 'spectrograph': 1, 'field': 'DARK', 'ccd': 1, 'filter': 'r', 'expTime': 300.0, 'dataType': 'DARK', 'taiObs': '2019-02-01T20:08:08', 'pfsDesignId': 1099528409104, 'slitOffset': 0.0}, tag=set())
dark.isr INFO: Converting exposure to floating point values
dark.isr INFO: Assembling CCD from amplifiers
dark.isr INFO: Masking defects.
dark.isr INFO: Masking NAN value pixels.
dark.isr INFO: Widening saturation trails.
dark.isr INFO: Interpolating masked pixels.
dark.isr WARN: No rough magnitude zero point set for filter r
dark.isr INFO: Setting rough magnitude zero point: 34.192803
dark.repair INFO: Identified 347 cosmic rays.
dark INFO: Scale on tiger2-sumire.princeton.edu:189253
CameraMapper INFO: Loading exposure registry from /projects/HSC/PFS/LAM/registry.sqlite3
CameraMapper INFO: Loading calib registry from /projects/HSC/PFS/LAM/CALIB/calibRegistry.sqlite3
dark INFO: Scatter combination
dark INFO: Combining {'arm': 'r', 'spectrograph': 1, 'ccd': 1, 'filter': 'NONE', 'calibTime': '2019-02-01T19:56:44.999998720', 'visit0': 11510, 'calibDate': '2019-02-01'} on tiger2-sumire.princeton.edu:189253
dark INFO: Writing {'arm': 'r', 'spectrograph': 1, 'ccd': 1, 'filter': 'NONE', 'calibTime': '2019-02-01T19:56:44.999998720', 'visit0': 11510, 'calibDate': '2019-02-01'} on tiger2-sumire.princeton.edu:189253

Comment by price [ 18/Jun/21 ]

Not sure whether this was user error or the pipeline has improved, but it doesn't seem like there's anything to do here.

Generated at Sat Feb 10 15:58:49 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.