Uploaded image for project: 'DRP 2-D Pipeline'
  1. DRP 2-D Pipeline
  2. PIPE2D-1138

ingestPfsImagesPgsql.py only partially succeeds when pfsConfig unavailable

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Done (View Workflow)
    • Priority: Normal
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Story Points:
      2
    • Sprint:
      PreEngRun10Feb

      Description

      See discussion on slack: https://sumire-pfs.slack.com/archives/C3AU3VCMU/p1672866688320709

      We've noticed that for certain visits, for example 84574..84578, that ingestion via ingestPfsImagesPgsql.py is problematic due to issues identifying the correct pfsDesign/pfsConfig.

      However, it has been noticed that some of the individual exposures have been ingested over, for example the b1 exposures for some visits, while the remainder b3, r1, r3 raw images were not.

      Arnaud reports for a test ingest with a list of visits:

      lsst-scipipe-3.0.0) [alefur@pfsa-usr01-gb ~]$ ingestPfsImagesPgsql.py /work/drp --pfsConfigDir /data/raw --mode=copy /data/raw/2023-01-04/sps/PFSA*32.fits
      /work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/argumentParser.py:782: FutureWarning: Gen2 Butler has been deprecated (PfsButler). It will be removed sometime after v23.0 but no earlier than the end of 2021.
      namespace.butler = dafPersist.Butler(outputs=outputs)
      /work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/argumentParser.py:782: FutureWarning: Gen2 Butler has been deprecated (PfsMapper). It will be removed sometime after v23.0 but no earlier than the end of 2021.
      namespace.butler = dafPersist.Butler(outputs=outputs)
      lsst.CameraMapper INFO: Loading exposure registry from /work/drp/registry.pgsql
      lsst.CameraMapper INFO: Loading calib registry from /work/drp/CALIB/calibRegistry.sqlite3

      That the first file in the list is ingested:

      lsst.ingestPfs INFO: /data/raw/2023-01-04/sps/PFSA08457432.fits -<copy>-> /work/drp/2023-01-04/PFSA084574r3.fits

      But then no further files are ingested after the pfsConfig is not identified:

      numexpr.utils INFO: Note: detected 128 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
      numexpr.utils INFO: Note: NumExpr detected 128 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
      lsst.ingestPfs WARN: Unable to find PfsConfig file for pfsDesignId=0x0000010001001010, visit=84574; using pfsDesign
      lsst.ingestPfs WARN: Failed to ingest file /data/raw/2023-01-04/sps/PFSA08457432.fits: Unable to find PfsConfig or PfsDesign for pfsDesignId=0x0000010001001010
      Traceback (most recent call last):
      File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_tasks/gb1d6de0934+6a8639994f/python/lsst/pipe/tasks/ingest.py", line 625, in run
      self.runFile(infile, registry, args, pos)
      File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/obs_pfs/w.2022.51/python/lsst/obs/pfs/ingest.py", line 750, in runFile
      self.ingestPfsConfig(pfsConfigDir, hduInfoList[0], args)
      File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/obs_pfs/w.2022.51/python/lsst/obs/pfs/ingest.py", line 625, in ingestPfsConfig
      infile = self.findPfsConfig(dirName, fileInfo)
      File "/work/stack_INFRA-312/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/obs_pfs/w.2022.51/python/lsst/obs/pfs/ingest.py", line 601, in findPfsConfig
      raise RuntimeError(f"Unable to find PfsConfig or PfsDesign for pfsDesignId=0x

      Unknown macro: {pfsDesignId}

      ")
      RuntimeError: Unable to find PfsConfig or PfsDesign for pfsDesignId=0x0000010001001010

      It should either fully ingest all the specified visits, or none. A partial success is confusing and incorrect. It seems that the pfsConfig check is not being applied prior to the first file being ingested.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hassan hassan
                Reporter:
                hassan hassan
                Reviewers:
                price
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: