[PIPE2D-1107] Integration test failing at defect ingestion Created: 03/Nov/22  Updated: 08/Feb/23  Resolved: 05/Nov/22

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Blocker
Reporter: hassan Assignee: price
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates
relates to PIPE2D-1094 Organise H4RG defects by logical seri... Done
Story Points: 2
Sprint: 2DDRP-2023 A
Reviewers: hassan

 Description   

Presently the integration tests (and most likely future weekly builds) fail. A snippet of the log output of running the tests against master are shown below.

I suspect that the problem is caused by the inclusion of PIPE2D-1094. The new script introduced in that ticket should be used.

   2114 + ingestCuratedCalibs.py /scratch/pprice/jenkins/integrationTest/461/INTEGRATION --calib /scratch/pprice/jenkins/integrationTest/461/INTEGRATION/CALIB /scratch/pprice/jenkin   2114 s/integrationTest/461/drp_pfs_data/curated/pfs/defects
   2115 root INFO: Loading config overrride file '/scratch/pprice/jenkins/integrationTest/461/obs_pfs/config/ingestCuratedCalibs.py'
   2116 lsst.CameraMapper INFO: Loading exposure registry from /scratch/pprice/jenkins/integrationTest/461/INTEGRATION/registry.sqlite3
   2117 Traceback (most recent call last):
   2118   File "/projects/HSC/PFS/stack/20220525/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_tasks/gb1d6de0934+6a8639994f/bin/ingestCuratedCalibs.py", line 3, in <module>
   2119     IngestCuratedCalibsTask.parseAndRun()
   2120   File "/projects/HSC/PFS/stack/20220525/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_tasks/gb1d6de0934+6a8639994f/python/lsst/pipe/tasks/ingest.py", line 431, in parseAnd   2120 Run
   2121     task.run(args)
   2122   File "/projects/HSC/PFS/stack/20220525/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_tasks/gb1d6de0934+6a8639994f/python/lsst/pipe/tasks/ingestCuratedCalibs.py", line 43,   2122  in run
   2123     calibs, calib_type = read_all(args.root, camera)
   2124   File "/projects/HSC/PFS/stack/20220525/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_tasks/gb1d6de0934+6a8639994f/python/lsst/pipe/tasks/read_curated_calibs.py", line 139   2124 , in read_all
   2125     raise RuntimeError(f"Detector {chip_name} not known to supplied camera "
   2126 RuntimeError: Detector sn1 not known to supplied camera PFS (examples: AG5,AG4,r4,AG1,b4,b2,AG6,AG2,m2,r2)


 Comments   
Comment by hassan [ 03/Nov/22 ]

Problem was indeed incorrect organization of H4RG defects. Updated integration test and weekly scripts to use new script to ingest defects.

Comment by hassan [ 03/Nov/22 ]

Integration test script still needs Gen3 section to be updated. Handing that to Paul.

Comment by price [ 04/Nov/22 ]

I changed a bunch of how this works.

  • The script is now called makePfsDefects, and it lives in drp_pfs_data.
  • The snX directories with the NIR defects are now in drp_pfs_data/nirDefects.
  • makePfsDefects builds the defects in drp_pfs_data/curated/pfs/defects, because that's what Gen3 requires.
  • makePfsDefects does not ingest the defects, because that's done differently for Gen2 and Gen3.
Comment by hassan [ 04/Nov/22 ]

Reviewed. All changes look fine, but shouldn't process_weekly.sh also be updated to use makePfsDefects?

Comment by price [ 05/Nov/22 ]

Added the change to the weekly, and merged.

Generated at Sat Feb 10 16:02:53 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.