https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-591 DRP 2-D Pipeline <p>Was trying to ingest post software migration images, but it failed in various places:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[pfs@shell2-ics drp]$ ingestPfsImages.py /data/drp/alefur --mode=link /data/raw/2020-05-27/sps/PFSA01774512.fits CameraMapper INFO: Loading exposure registry from /data/drp/alefur/registry.sqlite3 ingestPfs.parse WARN: Unable to find value for dataType (derived from IMAGETYP) ingestPfs.parse WARN: Unable to find value for expTime (derived from EXPTIME)  ingestPfs.parse WARN: Unable to find value for dateObs (derived from DATE-OBS) ingestPfs.parse WARN: Unable to find value for taiObs (derived from DATE-OBS) ingestPfs.parse WARN: translate_field failed to translate field: 'NoneType' object has no attribute 'strip' ingestPfs.parse WARN: No value set for pfsDesignId; using default (0x0000000000000000) ingestPfs.parse WARN: translate_date failed to translate dateObs: 'DATE-OBS not found' ingestPfs.parse WARN: translate_date failed to translate taiObs: 'DATE-OBS not found' Traceback (most recent call last): File "/software/drp-5.2/stack/miniconda3-4.5.12-1172c30/Linux64/obs_pfs/w.2020.21/python/lsst/obs/pfs/ingest.py", line 419, in runFile hduInfoList = super().runFile(infile, registry, args) File "/software/drp-5.2/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/18.1.0/python/lsst/pipe/tasks/ingest.py", line 533, in runFile outfile = self.parse.getDestination(args.butler, fileInfo, infile) File "/software/drp-5.2/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/18.1.0/python/lsst/pipe/tasks/ingest.py", line 189, in getDestination raw = butler.get("raw_filename", info)[0] File "/software/drp-5.2/stack/miniconda3-4.5.12-1172c30/Linux64/daf_persistence/18.1.0/python/lsst/daf/persistence/butler.py", line 1380, in get raise NoResults("No locations for get:", datasetType, dataId) lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:raw_filename dataId:DataId(initialdata={'site': 'S', 'category': 'A', 'visit': 17745, 'filter': 'r', 'arm': 'r', 'spectrograph': 1, 'ccd': 1, 'pfsDesignId': 0, 'slitOffset': 0.0, 'lamps': '', 'dither': -9999, 'shift': -9999, 'focus': -9999, 'attenuator': -9999, 'photodiode': -9999}, tag=set()) ingestPfs WARN: Failed to ingest file /data/raw/2020-05-27/sps/PFSA01774512.fits: No locations for get: datasetType:raw_filename dataId:DataId(initialdata={'site': 'S', 'category': 'A', 'visit': 17745, 'filter': 'r', 'arm': 'r', 'spectrograph': 1, 'ccd': 1, 'pfsDesignId': 0, 'slitOffset': 0.0, 'lamps': '', 'dither': -9999, 'shift': -9999, 'focus': -9999, 'attenuator': -9999, 'photodiode': -9999}, tag=set()) </pre> </div></div> PIPE2D-591

unable to ingest updated header images

Bug Blocker Done Done price arnaud.lefur Thu, 28 May 2020 20:37:58 +0000 Mon, 4 Jan 2021 20:23:18 +0000 Fri, 29 May 2020 18:39:54 +0000 0 3 <p>Looks like your image has a very sparse header. I'll see if I can find a copy.</p> <p>PHDU is the main header, 'image' has the image. Could add INHERIT even though that just papers things over. It papers things over for LSST, which is the consumer we care about.</p> <p>I successfully ingested this file (as part of <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-571" title="Ingest into a common data repo" class="issue-link" data-issue-key="PIPE2D-571"><del>PIPE2D-571</del></a>). Could you please try the <tt>w.2020.21</tt> release?</p> <p>Oh, I see that you are. So what's different between what you're doing and what I did?</p> <p>Ah, you've changed the report, and the responsible image to one I don't have a copy of.</p> <p>right sorry, I had pasted two extra lines</p> <p><a href="https://pfspipe.ipmu.jp/jira/browse/INSTRM-994" title="Add INHERIT to image HDU" class="issue-link" data-issue-key="INSTRM-994"><del>INSTRM-994</del></a> did actually fix most of this &#8211; the code had not actually been updated for 17787, but was for 17788. But there is one DRP issue in that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(lsst-scipipe-1172c30) -bash-4.2$ ingestPfsImages.py /data/drp/cpl --mode=link /data/raw/2020-05-29/sps/PFSA0177881* CameraMapper INFO: Loading exposure registry from /data/drp/cpl/registry.sqlite3 CameraMapper INFO: Loading calib registry from /data/drp/cpl/CALIB/calibRegistry.sqlite3 ingestPfs.parse WARN: Unable to find value for dataType (derived from IMAGETYP) </pre> </div></div> <p>That said, there are still problems. From guessed at the requirements in <tt>ingest.py</tt> and <tt>translator.py</tt> I will point out the following:</p> <p><tt>IMAGETYP</tt> will not/cannot be in the PFS headers. Please use <tt>DATA-TYP</tt> instead. I think I have supplied both for long time, but cannot add <tt>IMAGETYP</tt> any more. </p> <p><tt>ARM</tt> is now <tt>W_ARM</tt> and <tt>SPECNUM</tt> <tt>W_SPMOD</tt>. Subaru requirements.</p> <p>There is now an <tt>OBJECT</tt> card, although it will be "##NODATA##" (or whatever the gen2 simulator sends) until we get the instrument on the telescope.</p> <p>The headers in <tt>raw/2020-05-29/sps/PFSA0017788*.fits</tt> can be used to test against.</p> <p>Fixed the ingest.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> (lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-591 $ echo lsst.obs.pfs.PfsMapper &gt; _mapper (lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-591 $ ingestPfsImages.py . --pfsConfigDir /projects/HSC/PFS/Subaru/drp/pfsDesign --config parse.pfsDesignId=1099528409104 -- /projects/HSC/PFS/Subaru/raw/2020-05-29/sps/PFSA01778812.fits CameraMapper INFO: Loading exposure registry from /scratch/pprice/pipe2d-591/registry.sqlite3 ingestPfs INFO: /projects/HSC/PFS/Subaru/raw/2020-05-29/sps/PFSA01778812.fits --&lt;link&gt;--&gt; /scratch/pprice/pipe2d-591/2020-05-29/PFSA017788r1.fits ingestPfs INFO: /projects/HSC/PFS/Subaru/drp/pfsDesign/pfsConfig-0x0000010001001010-017788.fits --&lt;link&gt;--&gt; /scratch/pprice/pipe2d-591/pfsConfig/2020-05-29/pfsConfig-0x0000010001001010-017788.fits (lsst-scipipe) pprice@tiger2-sumire:/scratch/pprice/pipe2d-591 $ ingestPfsImages.py . --pfsConfigDir /projects/HSC/PFS/Subaru/drp/pfsDesign --config parse.pfsDesignId=1099528409104 -- /projects/HSC/PFS/Subaru/raw/2020-05-29/sps/PFSA01778811.fits CameraMapper INFO: Loading exposure registry from /scratch/pprice/pipe2d-591/registry.sqlite3 ingestPfs INFO: /projects/HSC/PFS/Subaru/raw/2020-05-29/sps/PFSA01778811.fits --&lt;link&gt;--&gt; /scratch/pprice/pipe2d-591/2020-05-29/PFSA017788b1.fits </pre> </div></div> <p>Merged to master.</p> Relates INSTRM-994 Development Rank 0|02qps2:2i Reviewers hassan Sprint 2DDRP-2021 A