-
Type: Story
-
Status: Open (View Workflow)
-
Priority: Normal
-
Resolution: Unresolved
-
Affects Version/s: None
-
Fix Version/s: None
-
Component/s: None
-
Labels:
Kiyoto Yabe reports:
I got the following errors recently:
(lsst-scipipe-7.0.1) [kiyoyabe@pfsa-usr02-gb reduction_new]$ pipetask --log-level .=INFO run --register-dataset-types -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/defaults -o u/kiyoyabe/pipe2d-1547/test/20241208b -p '$DRP_STELLA_DIR/pipelines/reduceExposure.yaml' -d "visit IN (115893, 115894)" --fail-fast -j 32 -c mergeArms:fitSkyModel.blockSize=20 py.warnings WARNING: /work/stack-2024-07-11/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/pipe_base/g8798d61f7d+6612571a14/python/lsst/pipe/base/graphBuilder.py:1673: FutureWarning: Querying for component datasets via Registry query methods is deprecated in favor of using DatasetRef and DatasetType methods on parent datasets. Only components=False will be supported after v26, and the components argument will be removed after v27. dataset_types = {ds.name: ds for ds in registry.queryDatasetTypes(all_names)} lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback: Traceback (most recent call last): File "/work/stack-2024-07-11/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 199, in run if (qgraph := script.qgraph(pipelineObj=pipeline, **kwargs, show=show)) is None: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/work/stack-2024-07-11/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 210, in qgraph qgraph = f.makeGraph(pipelineObj, args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/work/stack-2024-07-11/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 622, in makeGraph qgraph = graphBuilder.makeGraph( ^^^^^^^^^^^^^^^^^^^^^^^ File "/work/stack-2024-07-11/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/pipe_base/g8798d61f7d+6612571a14/python/lsst/pipe/base/graphBuilder.py", line 1840, in makeGraph return scaffolding.makeQuantumGraph( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/work/stack-2024-07-11/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/pipe_base/g8798d61f7d+6612571a14/python/lsst/pipe/base/graphBuilder.py", line 1643, in makeQuantumGraph graph = QuantumGraph( ^^^^^^^^^^^^^ File "/work/stack-2024-07-11/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/pipe_base/g8798d61f7d+6612571a14/python/lsst/pipe/base/graph/graph.py", line 147, in __init__ self._buildGraphs( File "/work/stack-2024-07-11/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/pipe_base/g8798d61f7d+6612571a14/python/lsst/pipe/base/graph/graph.py", line 246, in _buildGraphs self._datasetRefDict.addProducer(dsRef, value) File "/work/stack-2024-07-11/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/pipe_base/g8798d61f7d+6612571a14/python/lsst/pipe/base/graph/_implDetails.py", line 83, in addProducer raise ValueError(f"Only one node is allowed to produce {key}, the current producer is {existing}") ValueError: Only one node is allowed to produce calexp@{instrument: 'PFS', arm: 'r', spectrograph: 4, visit: 115894, ...} [sc=Exposure] (run=u/kiyoyabe/pipe2d-1547/test/20241208b/20241209T060306Z id=021ff1e6-e78a-46ed-9857-0507eaff0931), the current producer is QuantumNode(quantum=Quantum(taskName=pfs.drp.stella.cosmicray.CompareCosmicRayTask, dataId={instrument: 'PFS', arm: 'r', spectrograph: 4, visit_group: 115894}), taskDef=TaskDef(pfs.drp.stella.cosmicray.CompareCosmicRayTask, label=cosmicray2), nodeId=49b7245a-6d2e-4de4-9637-4da30758c603)I made a mistake the other day and ran defineVisitGroup.py for some visits (including 115894) as a single visit set, and I suspect that is the root cause, but is it the case?
drp=> select * from visit_group_join where visit=115894; instrument | visit_group | visit ------------+-------------+-------- PFS | 115893 | 115894 PFS | 115894 | 115894 (2 rows)115894 is in two different visit_groups on the database.
The database should not allow a visit to have more than one visit_group. This is probably a bug in the dimensions.yaml.
- relates to
-
PIPE2D-1606 Automatically identify visit groups
- Open