[PIPE2D-1138] ingestPfsImagesPgsql.py only partially succeeds when pfsConfig unavailable

Relates — Mon, 30 Jan 2023 21:40:51 +0000

See discussion on slack: https://sumire-pfs.slack.com/archives/C3AU3VCMU/p1672866688320709

We've noticed that for certain visits, for example 84574..84578, that ingestion via ingestPfsImagesPgsql.py is problematic due to issues identifying the correct pfsDesign/pfsConfig.

However, it has been noticed that some of the individual exposures have been ingested over, for example the b1 exposures for some visits, while the remainder b3, r1, r3 raw images were not.

Arnaud reports for a test ingest with a list of visits:

lsst-scipipe-3.0.0) [alefur@pfsa-usr01-gb ~]$ ingestPfsImagesPgsql.py /work/drp --pfsConfigDir /data/raw --mode=copy /data/raw/2023-01-04/sps/PFSA*32.fits
/work/stack_~~INFRA-312~~/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/argumentParser.py:782: FutureWarning: Gen2 Butler has been deprecated (PfsButler). It will be removed sometime after v23.0 but no earlier than the end of 2021.
namespace.butler = dafPersist.Butler(outputs=outputs)
/work/stack_~~INFRA-312~~/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/argumentParser.py:782: FutureWarning: Gen2 Butler has been deprecated (PfsMapper). It will be removed sometime after v23.0 but no earlier than the end of 2021.
namespace.butler = dafPersist.Butler(outputs=outputs)
lsst.CameraMapper INFO: Loading exposure registry from /work/drp/registry.pgsql
lsst.CameraMapper INFO: Loading calib registry from /work/drp/CALIB/calibRegistry.sqlite3

That the first file in the list is ingested:

lsst.ingestPfs INFO: /data/raw/2023-01-04/sps/PFSA08457432.fits -~~<copy>~~-> /work/drp/2023-01-04/PFSA084574r3.fits

But then no further files are ingested after the pfsConfig is not identified:

numexpr.utils INFO: Note: detected 128 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
numexpr.utils INFO: Note: NumExpr detected 128 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
lsst.ingestPfs WARN: Unable to find PfsConfig file for pfsDesignId=0x0000010001001010, visit=84574; using pfsDesign
lsst.ingestPfs WARN: Failed to ingest file /data/raw/2023-01-04/sps/PFSA08457432.fits: Unable to find PfsConfig or PfsDesign for pfsDesignId=0x0000010001001010
Traceback (most recent call last):
File "/work/stack_~~INFRA-312~~/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_tasks/gb1d6de0934+6a8639994f/python/lsst/pipe/tasks/ingest.py", line 625, in run
self.runFile(infile, registry, args, pos)
File "/work/stack_~~INFRA-312~~/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/obs_pfs/w.2022.51/python/lsst/obs/pfs/ingest.py", line 750, in runFile
self.ingestPfsConfig(pfsConfigDir, hduInfoList[0], args)
File "/work/stack_~~INFRA-312~~/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/obs_pfs/w.2022.51/python/lsst/obs/pfs/ingest.py", line 625, in ingestPfsConfig
infile = self.findPfsConfig(dirName, fileInfo)
File "/work/stack_~~INFRA-312~~/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/obs_pfs/w.2022.51/python/lsst/obs/pfs/ingest.py", line 601, in findPfsConfig
raise RuntimeError(f"Unable to find PfsConfig or PfsDesign for pfsDesignId=0x

Unknown macro: {pfsDesignId}

")
RuntimeError: Unable to find PfsConfig or PfsDesign for pfsDesignId=0x0000010001001010

It should either fully ingest all the specified visits, or none. A partial success is confusing and incorrect. It seems that the pfsConfig check is not being applied prior to the first file being ingested.

PFS-JIRA

[PIPE2D-1138] ingestPfsImagesPgsql.py only partially succeeds when pfsConfig unavailable