[PIPE2D-619] Support multiple calibs on a single day Created: 23/Jul/20  Updated: 05/Jan/21  Resolved: 25/Sep/20

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: price Assignee: price
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Story Points: 3
Sprint: 2DDRP-2021 A
Reviewers: hassan

 Description   

We need to be able to support the use of multiple calibs on a single day.

That probably means using a date+time instead of just date in dateObs and calibDate. Calib ingestion may need some small tweaks to work with this.

 Comments   
Comment by price [ 31/Jul/20 ]

See also DM-26171 ("calibrations cannot be retrieved on the last day of a validity range").

Comment by price [ 27/Aug/20 ]

This requires changing both the raw and calib registries, and so will require re-ingesting everything.

Comment by price [ 28/Aug/20 ]

This is working now. hassan, could you please review?

Comment by price [ 28/Aug/20 ] 
Changes required:
* calibDate in calib generation
* calibDate in calib ingestion
* obsDate in raw ingestion
* Calib lookup in Mapper

Let's validate using the PIPE2D-618 dataset, where we had to play games with the calib registry; such games should not be required now. We need to ingest everything from scratch, to include the times in the registry.

(lsst-scipipe) pprice@tiger2-sumire:/tigress/pprice/pipe2d-619 $ mkdir DATA
(lsst-scipipe) pprice@tiger2-sumire:/tigress/pprice/pipe2d-619 $ echo lsst.obs.pfs.PfsMapper > DATA/_mapper
(lsst-scipipe) pprice@tiger2-sumire:/tigress/pprice/pipe2d-619 $ ingestPfsImages.py DATA --pfsConfigDir /projects/HSC/PFS/Subaru/drp/pfsDesign --config parse.pfsDesignId=1099528409104 --ignore-ingested -- '/projects/HSC/PFS/Subaru/raw/2020-05-*/sps/PFSA*.fits' '/projects/HSC/PFS/Subaru/raw/2020-06-*/sps/PFSA*.fits'
(lsst-scipipe) pprice@tiger2-sumire:/tigress/pprice/pipe2d-619 $ mkdir CALIB
(lsst-scipipe) pprice@tiger2-sumire:/tigress/pprice/pipe2d-619 $ ingestPfsCalibs.py DATA --calib CALIB --validity=1800 --doraise --mode=copy ~/pfs/drp_pfs_data/detectorMap/detectorMap-2019Jul-*.fits
(lsst-scipipe) pprice@tiger2-sumire:/tigress/pprice/pipe2d-619 $ generateCommands.py DATA --calib CALIB --rerun calibs --blocks=pipe2d_618 -j 20 ../pipe2d-618/pipe2d-618.yaml calibs.sh
generateCommands INFO: Start writing shell commands on 'calibs.sh'
generateCommands INFO: Processing calib block 'pipe2d_618'
generateCommands INFO: End writing shell commands on 'calibs.sh'
(lsst-scipipe) pprice@tiger2-sumire:/tigress/pprice/pipe2d-619 $ bash calibs.sh

constructFiberTrace.py DATA --calib=CALIB --rerun=fiberTrace --doraise --batch-type=smp --cores=10 --id visit=18228..18231
constructFiberTrace.py DATA --calib=CALIB --rerun=fiberTrace --doraise --batch-type=smp --cores=10 --id visit=18236..18238
constructFiberTrace.py DATA --calib=CALIB --rerun=fiberTrace --doraise --batch-type=smp --cores=10 --id visit=18245..18248
...

(lsst-scipipe) pprice@tiger2-sumire:/tigress/pprice/pipe2d-619 $ ingestPfsCalibs.py DATA --calib CALIB --validity 1800 --doraise --mode=copy DATA/rerun/fiberTrace/FIBERTRACE/pfsFiberTrace-*.fits

>>> from lsst.daf.persistence import Butler
>>> butler = Butler("DATA", calibRoot="/tigress/pprice/pipe2d-619/CALIB-test")
>>> butler.get("fiberTrace_filename", visit=18228, arm="r")[0]
'/tigress/pprice/pipe2d-619/CALIB-test/FIBERTRACE/pfsFiberTrace-2020-06-02-018228-r1.fits'
>>> butler.get("fiberTrace_filename", visit=18236, arm="r")[0]
'/tigress/pprice/pipe2d-619/CALIB-test/FIBERTRACE/pfsFiberTrace-2020-06-02-018236-r1.fits'
Comment by price [ 25/Sep/20 ]

Merged to master.

Generated at Sat Feb 10 15:55:29 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.