[PIPE2D-591] unable to ingest updated header images Created: 29/May/20  Updated: 05/Jan/21  Resolved: 30/May/20

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Blocker
Reporter: arnaud.lefur Assignee: price
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates
relates to INSTRM-994 Add INHERIT to image HDU Done
Sprint: 2DDRP-2021 A
Reviewers: hassan

 Description   

Was trying to ingest post software migration images, but it failed in various places:

[pfs@shell2-ics drp]$ ingestPfsImages.py /data/drp/alefur --mode=link /data/raw/2020-05-27/sps/PFSA01774512.fits CameraMapper INFO: Loading exposure registry from /data/drp/alefur/registry.sqlite3 
ingestPfs.parse WARN: Unable to find value for dataType (derived from IMAGETYP) 
ingestPfs.parse WARN: Unable to find value for expTime (derived from EXPTIME) 
ingestPfs.parse WARN: Unable to find value for dateObs (derived from DATE-OBS) 
ingestPfs.parse WARN: Unable to find value for taiObs (derived from DATE-OBS) 
ingestPfs.parse WARN: translate_field failed to translate field: 'NoneType' object has no attribute 'strip' 
ingestPfs.parse WARN: No value set for pfsDesignId; using default (0x0000000000000000) 
ingestPfs.parse WARN: translate_date failed to translate dateObs: 'DATE-OBS not found' 
ingestPfs.parse WARN: translate_date failed to translate taiObs: 'DATE-OBS not found' 
Traceback (most recent call last): 
 File "/software/drp-5.2/stack/miniconda3-4.5.12-1172c30/Linux64/obs_pfs/w.2020.21/python/lsst/obs/pfs/ingest.py", line 419, in runFile 
 hduInfoList = super().runFile(infile, registry, args) 
 File "/software/drp-5.2/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/18.1.0/python/lsst/pipe/tasks/ingest.py", line 533, in runFile 
 outfile = self.parse.getDestination(args.butler, fileInfo, infile) 
 File "/software/drp-5.2/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/18.1.0/python/lsst/pipe/tasks/ingest.py", line 189, in getDestination 
 raw = butler.get("raw_filename", info)[0] 
 File "/software/drp-5.2/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/butler.py", line 1380, in get 
 raise NoResults("No locations for get:", datasetType, dataId) 
lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:raw_filename dataId:DataId(initialdata={'site': 'S', 'category': 'A', 'visit': 17745, 'filter': 'r', 'arm': 'r', 'spectrograph': 1, 'ccd': 1, 'pfsDesignId': 0, 'slitOffset': 0.0, 'lamps': '', 'dither': -9999, 'shift': -9999, 'focus': -9999, 'attenuator': -9999, 'photodiode': -9999}, tag=set()) 
ingestPfs WARN: Failed to ingest file /data/raw/2020-05-27/sps/PFSA01774512.fits: No locations for get: datasetType:raw_filename dataId:DataId(initialdata={'site': 'S', 'category': 'A', 'visit': 17745, 'filter': 'r', 'arm': 'r', 'spectrograph': 1, 'ccd': 1, 'pfsDesignId': 0, 'slitOffset': 0.0, 'lamps': '', 'dither': -9999, 'shift': -9999, 'focus': -9999, 'attenuator': -9999, 'photodiode': -9999}, tag=set())


 Comments   
Comment by price [ 29/May/20 ]

Looks like your image has a very sparse header. I'll see if I can find a copy.

Comment by cloomis [ 29/May/20 ]

PHDU is the main header, 'image' has the image. Could add INHERIT even though that just papers things over. It papers things over for LSST, which is the consumer we care about.

Comment by price [ 29/May/20 ]

I successfully ingested this file (as part of PIPE2D-571). Could you please try the w.2020.21 release?

Comment by price [ 29/May/20 ]

Oh, I see that you are. So what's different between what you're doing and what I did?

Comment by price [ 29/May/20 ]

Ah, you've changed the report, and the responsible image to one I don't have a copy of.

Comment by arnaud.lefur [ 29/May/20 ]

right sorry, I had pasted two extra lines

Comment by cloomis [ 29/May/20 ]

INSTRM-994 did actually fix most of this – the code had not actually been updated for 17787, but was for 17788. But there is one DRP issue in that:

(lsst-scipipe-1172c30) -bash-4.2$ ingestPfsImages.py /data/drp/cpl --mode=link /data/raw/2020-05-29/sps/PFSA0177881*
CameraMapper INFO: Loading exposure registry from /data/drp/cpl/registry.sqlite3
CameraMapper INFO: Loading calib registry from /data/drp/cpl/CALIB/calibRegistry.sqlite3
ingestPfs.parse WARN: Unable to find value for dataType (derived from IMAGETYP)

That said, there are still problems. From guessed at the requirements in ingest.py and translator.py I will point out the following:

IMAGETYP will not/cannot be in the PFS headers. Please use DATA-TYP instead. I think I have supplied both for long time, but cannot add IMAGETYP any more.

ARM is now W_ARM and SPECNUM W_SPMOD. Subaru requirements.

There is now an OBJECT card, although it will be "##NODATA##" (or whatever the gen2 simulator sends) until we get the instrument on the telescope.

The headers in raw/2020-05-29/sps/PFSA0017788*.fits can be used to test against.

Comment by price [ 30/May/20 ]

Fixed the ingest.

(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-591 $ echo lsst.obs.pfs.PfsMapper > _mapper
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-591 $ ingestPfsImages.py . --pfsConfigDir /projects/HSC/PFS/Subaru/drp/pfsDesign --config parse.pfsDesignId=1099528409104 -- /projects/HSC/PFS/Subaru/raw/2020-05-29/sps/PFSA01778812.fits 
CameraMapper INFO: Loading exposure registry from /scratch/pprice/pipe2d-591/registry.sqlite3
ingestPfs INFO: /projects/HSC/PFS/Subaru/raw/2020-05-29/sps/PFSA01778812.fits --<link>--> /scratch/pprice/pipe2d-591/2020-05-29/PFSA017788r1.fits
ingestPfs INFO: /projects/HSC/PFS/Subaru/drp/pfsDesign/pfsConfig-0x0000010001001010-017788.fits --<link>--> /scratch/pprice/pipe2d-591/pfsConfig/2020-05-29/pfsConfig-0x0000010001001010-017788.fits
(lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-591 $ ingestPfsImages.py . --pfsConfigDir /projects/HSC/PFS/Subaru/drp/pfsDesign --config parse.pfsDesignId=1099528409104 -- /projects/HSC/PFS/Subaru/raw/2020-05-29/sps/PFSA01778811.fits 
CameraMapper INFO: Loading exposure registry from /scratch/pprice/pipe2d-591/registry.sqlite3
ingestPfs INFO: /projects/HSC/PFS/Subaru/raw/2020-05-29/sps/PFSA01778811.fits --<link>--> /scratch/pprice/pipe2d-591/2020-05-29/PFSA017788b1.fits
Comment by price [ 30/May/20 ]

Merged to master.

Generated at Fri Apr 25 23:11:33 JST 2025 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.