[PIPE2D-1095] Fix w.2022.42 weekly test failure Created: 17/Oct/22  Updated: 08/Feb/23  Resolved: 21/Oct/22

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Normal
Reporter: hassan Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates
relates to PIPE2D-668 Support ISR processing for NIR data Done
relates to PIPE2D-1088 Add PFSB-compliant NIR data to weeklies Done
Story Points: 1
Sprint: 2DDRP-2023 A
Reviewers: price

 Description   

The latest weekly, w.2022.42 fails following the changes introduced by PIPE2D-668.

A snippet of the log is provided below.

+ sh /scratch/pprice/jenkins/weekly/2022-10-16/process/science.sh
+ reduceExposure.py /scratch/pprice/jenkins/weekly/2022-10-16/process --calib=/scratch/pprice/jenkins/weekly/2022-10-16/process/CALIB --rerun=science/pipeline --doraise -j10 --id 'visit=1000^1001^1004^1005'
root INFO: Loading config overrride file '/scratch/pprice/jenkins/weekly/2022-10-16/build/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/obs_pfs/w.2022.42/config/reduceExposure.py'
lsst.CameraMapper INFO: Loading exposure registry from /scratch/pprice/jenkins/weekly/2022-10-16/process/registry.sqlite3
lsst.CameraMapper INFO: Loading calib registry from /scratch/pprice/jenkins/weekly/2022-10-16/process/CALIB/calibRegistry.sqlite3
lsst.CameraMapper INFO: Loading calib registry from /scratch/pprice/jenkins/weekly/2022-10-16/process/CALIB/calibRegistry.sqlite3
Traceback (most recent call last):
File "/scratch/pprice/jenkins/weekly/2022-10-16/build/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/drp_stella/w.2022.42/bin/reduceExposure.py", line 25, in <module>
ReduceExposureTask.parseAndRun()
File "/scratch/pprice/jenkins/weekly/2022-10-16/build/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py", line 682, in parseAndRun
parsedCmd = argumentParser.parse_args(config=config, args=args, log=log, override=cls.applyOverrides)
File "/scratch/pprice/jenkins/weekly/2022-10-16/build/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/argumentParser.py", line 788, in parse_args
self._processDataIds(namespace)
File "/scratch/pprice/jenkins/weekly/2022-10-16/build/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/argumentParser.py", line 901, in _processDataIds
dataIdContainer.makeDataRefList(namespace)
File "/scratch/pprice/jenkins/weekly/2022-10-16/build/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/argumentParser.py", line 203, in makeDataRefList
refList = dafPersist.searchDataRefs(
File "/scratch/pprice/jenkins/weekly/2022-10-16/build/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/daf_persistence/g6a31054a6e+4ae6138134/python/lsst/daf/persistence/butlerHelpers.py", line 57, in searchDataRefs
return [dr for dr in refList if dataExists(dr)]
File "/scratch/pprice/jenkins/weekly/2022-10-16/build/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/daf_persistence/g6a31054a6e+4ae6138134/python/lsst/daf/persistence/butlerHelpers.py", line 57, in <listcomp>
return [dr for dr in refList if dataExists(dr)]
File "/scratch/pprice/jenkins/weekly/2022-10-16/build/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/daf_persistence/g6a31054a6e+4ae6138134/python/lsst/daf/persistence/butlerHelpers.py", line 80, in dataExists
return dataRef.datasetExists()
File "/scratch/pprice/jenkins/weekly/2022-10-16/build/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/daf_persistence/g6a31054a6e+4ae6138134/python/lsst/daf/persistence/butlerSubset.py", line 313, in datasetExists
return self.butlerSubset.butler.datasetExists(
File "/scratch/pprice/jenkins/weekly/2022-10-16/build/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/daf_persistence/g6a31054a6e+4ae6138134/python/lsst/daf/persistence/butler.py", line 1261, in datasetExists
locations = self._locate(datasetType, dataId, write=write)
File "/scratch/pprice/jenkins/weekly/2022-10-16/build/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/daf_persistence/g6a31054a6e+4ae6138134/python/lsst/daf/persistence/butler.py", line 1347, in _locate
bypass = bypass()
File "/scratch/pprice/jenkins/weekly/2022-10-16/build/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/daf_persistence/g6a31054a6e+4ae6138134/python/lsst/daf/persistence/butler.py", line 1375, in <lambda>
return lambda: bypassFunc(location.datasetType, pythonType, location, dataId)
File "/scratch/pprice/jenkins/weekly/2022-10-16/build/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/obs_pfs/w.2022.42/python/lsst/obs/pfs/pfsMapper.py", line 260, in bypass_raw
hasResetFrame = md["W_4FMTVR"] >= 3 # extra IMG/REF HDUs for the initial reset frame
File "/scratch/pprice/jenkins/weekly/2022-10-16/build/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/daf_base/gcc072456be+b6f9ac6643/python/lsst/daf/base/propertyContainer/propertyContainerContinued.py", line 727, in _getitem_
return self.getScalar(name)
File "/scratch/pprice/jenkins/weekly/2022-10-16/build/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/daf_base/gcc072456be+b6f9ac6643/python/lsst/daf/base/propertyContainer/propertyContainerContinued.py", line 878, in getScalar
return _propertyContainerGet(self, name, returnStyle=ReturnStyle.SCALAR)
File "/scratch/pprice/jenkins/weekly/2022-10-16/build/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/daf_base/gcc072456be+b6f9ac6643/python/lsst/daf/base/propertyContainer/propertyContainerContinued.py", line 215, in _propertyContainerGet
raise KeyError(name + " not found")
KeyError: 'W_4FMTVR not found'

Investigate and fix.

 Comments   
Comment by hassan [ 17/Oct/22 ]

The problem is that the pfs_pipe2d script process_science.sh was not tested. In addition to changes made in PIPE2D-668 to ignore n band data (which currently is based on simulations and are not in the PFSB format which is the case for real n band data, and is the required format following PIPE2D-668) updates to the science.yaml script to ignore n data are also necessary.

Comment by hassan [ 21/Oct/22 ]

Merged to master.

Generated at Sat Feb 10 16:02:38 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.