-
Type:
Bug
-
Status: Done (View Workflow)
-
Priority:
Blocker
-
Resolution: Done
-
Affects Version/s: None
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Sprint:2DDRP-2021 A
Was trying to ingest post software migration images, but it failed in various places:
[pfs@shell2-ics drp]$ ingestPfsImages.py /data/drp/alefur --mode=link /data/raw/2020-05-27/sps/PFSA01774512.fits CameraMapper INFO: Loading exposure registry from /data/drp/alefur/registry.sqlite3
ingestPfs.parse WARN: Unable to find value for dataType (derived from IMAGETYP)
ingestPfs.parse WARN: Unable to find value for expTime (derived from EXPTIME)
ingestPfs.parse WARN: Unable to find value for dateObs (derived from DATE-OBS)
ingestPfs.parse WARN: Unable to find value for taiObs (derived from DATE-OBS)
ingestPfs.parse WARN: translate_field failed to translate field: 'NoneType' object has no attribute 'strip'
ingestPfs.parse WARN: No value set for pfsDesignId; using default (0x0000000000000000)
ingestPfs.parse WARN: translate_date failed to translate dateObs: 'DATE-OBS not found'
ingestPfs.parse WARN: translate_date failed to translate taiObs: 'DATE-OBS not found'
Traceback (most recent call last):
File "/software/drp-5.2/stack/miniconda3-4.5.12-1172c30/Linux64/obs_pfs/w.2020.21/python/lsst/obs/pfs/ingest.py", line 419, in runFile
hduInfoList = super().runFile(infile, registry, args)
File "/software/drp-5.2/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/18.1.0/python/lsst/pipe/tasks/ingest.py", line 533, in runFile
outfile = self.parse.getDestination(args.butler, fileInfo, infile)
File "/software/drp-5.2/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/18.1.0/python/lsst/pipe/tasks/ingest.py", line 189, in getDestination
raw = butler.get("raw_filename", info)[0]
File "/software/drp-5.2/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/butler.py", line 1380, in get
raise NoResults("No locations for get:", datasetType, dataId)
lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:raw_filename dataId:DataId(initialdata={'site': 'S', 'category': 'A', 'visit': 17745, 'filter': 'r', 'arm': 'r', 'spectrograph': 1, 'ccd': 1, 'pfsDesignId': 0, 'slitOffset': 0.0, 'lamps': '', 'dither': -9999, 'shift': -9999, 'focus': -9999, 'attenuator': -9999, 'photodiode': -9999}, tag=set())
ingestPfs WARN: Failed to ingest file /data/raw/2020-05-27/sps/PFSA01774512.fits: No locations for get: datasetType:raw_filename dataId:DataId(initialdata={'site': 'S', 'category': 'A', 'visit': 17745, 'filter': 'r', 'arm': 'r', 'spectrograph': 1, 'ccd': 1, 'pfsDesignId': 0, 'slitOffset': 0.0, 'lamps': '', 'dither': -9999, 'shift': -9999, 'focus': -9999, 'attenuator': -9999, 'photodiode': -9999}, tag=set())
- relates to
-
INSTRM-994 Add INHERIT to image HDU
-
- Done
-
right sorry, I had pasted two extra lines
INSTRM-994 did actually fix most of this – the code had not actually been updated for 17787, but was for 17788. But there is one DRP issue in that:
(lsst-scipipe-1172c30) -bash-4.2$ ingestPfsImages.py /data/drp/cpl --mode=link /data/raw/2020-05-29/sps/PFSA0177881* CameraMapper INFO: Loading exposure registry from /data/drp/cpl/registry.sqlite3 CameraMapper INFO: Loading calib registry from /data/drp/cpl/CALIB/calibRegistry.sqlite3 ingestPfs.parse WARN: Unable to find value for dataType (derived from IMAGETYP)
That said, there are still problems. From guessed at the requirements in ingest.py and translator.py I will point out the following:
IMAGETYP will not/cannot be in the PFS headers. Please use DATA-TYP instead. I think I have supplied both for long time, but cannot add IMAGETYP any more.
ARM is now W_ARM and SPECNUM W_SPMOD. Subaru requirements.
There is now an OBJECT card, although it will be "##NODATA##" (or whatever the gen2 simulator sends) until we get the instrument on the telescope.
The headers in raw/2020-05-29/sps/PFSA0017788*.fits can be used to test against.
Fixed the ingest.
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-591 $ echo lsst.obs.pfs.PfsMapper > _mapper (lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-591 $ ingestPfsImages.py . --pfsConfigDir /projects/HSC/PFS/Subaru/drp/pfsDesign --config parse.pfsDesignId=1099528409104 -- /projects/HSC/PFS/Subaru/raw/2020-05-29/sps/PFSA01778812.fits CameraMapper INFO: Loading exposure registry from /scratch/pprice/pipe2d-591/registry.sqlite3 ingestPfs INFO: /projects/HSC/PFS/Subaru/raw/2020-05-29/sps/PFSA01778812.fits --<link>--> /scratch/pprice/pipe2d-591/2020-05-29/PFSA017788r1.fits ingestPfs INFO: /projects/HSC/PFS/Subaru/drp/pfsDesign/pfsConfig-0x0000010001001010-017788.fits --<link>--> /scratch/pprice/pipe2d-591/pfsConfig/2020-05-29/pfsConfig-0x0000010001001010-017788.fits (lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-591 $ ingestPfsImages.py . --pfsConfigDir /projects/HSC/PFS/Subaru/drp/pfsDesign --config parse.pfsDesignId=1099528409104 -- /projects/HSC/PFS/Subaru/raw/2020-05-29/sps/PFSA01778811.fits CameraMapper INFO: Loading exposure registry from /scratch/pprice/pipe2d-591/registry.sqlite3 ingestPfs INFO: /projects/HSC/PFS/Subaru/raw/2020-05-29/sps/PFSA01778811.fits --<link>--> /scratch/pprice/pipe2d-591/2020-05-29/PFSA017788b1.fits
Ah, you've changed the report, and the responsible image to one I don't have a copy of.