Uploaded image for project: 'DRP 2-D Pipeline'
  1. DRP 2-D Pipeline
  2. PIPE2D-1601

Database allows visits to have multiple visit_groups

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Open (View Workflow)
    • Priority: Normal
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      Kiyoto Yabe reports:

      I got the following errors recently:

      (lsst-scipipe-7.0.1) [kiyoyabe@pfsa-usr02-gb reduction_new]$ pipetask --log-level .=INFO run --register-dataset-types -b /work/datastore --instrument lsst.obs.pfs.PrimeFocusSpectrograph -i PFS/defaults -o u/kiyoyabe/pipe2d-1547/test/20241208b -p '$DRP_STELLA_DIR/pipelines/reduceExposure.yaml' -d "visit IN (115893, 115894)" --fail-fast -j 32 -c mergeArms:fitSkyModel.blockSize=20
      py.warnings WARNING: /work/stack-2024-07-11/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/pipe_base/g8798d61f7d+6612571a14/python/lsst/pipe/base/graphBuilder.py:1673: FutureWarning: Querying for component datasets via Registry query methods is deprecated in favor of using DatasetRef and DatasetType methods on parent datasets. Only components=False will be supported after v26, and the components argument will be removed after v27.
        dataset_types = {ds.name: ds for ds in registry.queryDatasetTypes(all_names)}
      
      lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
      Traceback (most recent call last):
        File "/work/stack-2024-07-11/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 199, in run
          if (qgraph := script.qgraph(pipelineObj=pipeline, **kwargs, show=show)) is None:
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/work/stack-2024-07-11/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 210, in qgraph
          qgraph = f.makeGraph(pipelineObj, args)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/work/stack-2024-07-11/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/ctrl_mpexec/g218a3a8f53+ca4789321c/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 622, in makeGraph
          qgraph = graphBuilder.makeGraph(
                   ^^^^^^^^^^^^^^^^^^^^^^^
        File "/work/stack-2024-07-11/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/pipe_base/g8798d61f7d+6612571a14/python/lsst/pipe/base/graphBuilder.py", line 1840, in makeGraph
          return scaffolding.makeQuantumGraph(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/work/stack-2024-07-11/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/pipe_base/g8798d61f7d+6612571a14/python/lsst/pipe/base/graphBuilder.py", line 1643, in makeQuantumGraph
          graph = QuantumGraph(
                  ^^^^^^^^^^^^^
        File "/work/stack-2024-07-11/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/pipe_base/g8798d61f7d+6612571a14/python/lsst/pipe/base/graph/graph.py", line 147, in __init__
          self._buildGraphs(
        File "/work/stack-2024-07-11/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/pipe_base/g8798d61f7d+6612571a14/python/lsst/pipe/base/graph/graph.py", line 246, in _buildGraphs
          self._datasetRefDict.addProducer(dsRef, value)
        File "/work/stack-2024-07-11/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/pipe_base/g8798d61f7d+6612571a14/python/lsst/pipe/base/graph/_implDetails.py", line 83, in addProducer
          raise ValueError(f"Only one node is allowed to produce {key}, the current producer is {existing}")
      ValueError: Only one node is allowed to produce calexp@{instrument: 'PFS', arm: 'r', spectrograph: 4, visit: 115894, ...} [sc=Exposure] (run=u/kiyoyabe/pipe2d-1547/test/20241208b/20241209T060306Z id=021ff1e6-e78a-46ed-9857-0507eaff0931), the current producer is QuantumNode(quantum=Quantum(taskName=pfs.drp.stella.cosmicray.CompareCosmicRayTask, dataId={instrument: 'PFS', arm: 'r', spectrograph: 4, visit_group: 115894}), taskDef=TaskDef(pfs.drp.stella.cosmicray.CompareCosmicRayTask, label=cosmicray2), nodeId=49b7245a-6d2e-4de4-9637-4da30758c603)
      

      I made a mistake the other day and ran defineVisitGroup.py for some visits (including 115894) as a single visit set, and I suspect that is the root cause, but is it the case?

      drp=> select * from visit_group_join where visit=115894;
       instrument | visit_group | visit
      ------------+-------------+--------
       PFS        |      115893 | 115894
       PFS        |      115894 | 115894
      (2 rows)
      

      115894 is in two different visit_groups on the database.

      The database should not allow a visit to have more than one visit_group. This is probably a bug in the dimensions.yaml.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                price price
                Reporter:
                price price
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: