https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-619 DRP 2-D Pipeline <p>We need to be able to support the use of multiple calibs on a single day.</p> <p>That probably means using a date+time instead of just date in <tt>dateObs</tt> and <tt>calibDate</tt>. Calib ingestion may need some small tweaks to work with this.</p> PIPE2D-619

Support multiple calibs on a single day

Story Normal Done Done price price Wed, 22 Jul 2020 19:45:25 +0000 Mon, 4 Jan 2021 20:23:20 +0000 Thu, 24 Sep 2020 22:13:25 +0000 0 1 <p>See also <a href="http://jira.lsstcorp.org/browse/DM-26171" class="external-link" rel="nofollow">DM-26171</a> ("calibrations cannot be retrieved on the last day of a validity range").</p> <p>This requires changing both the raw and calib registries, and so will require re-ingesting everything.</p> <p>This is working now. <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=hassan" class="user-hover" rel="hassan">hassan</a>, could you please review?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Changes required: * calibDate in calib generation * calibDate in calib ingestion * obsDate in raw ingestion * Calib lookup in Mapper Let's validate using the PIPE2D-618 dataset, where we had to play games with the calib registry; such games should not be required now. We need to ingest everything from scratch, to include the times in the registry. (lsst-scipipe) pprice@tiger2-sumire:/tigress/pprice/pipe2d-619 $ mkdir DATA (lsst-scipipe) pprice@tiger2-sumire:/tigress/pprice/pipe2d-619 $ echo lsst.obs.pfs.PfsMapper &gt; DATA/_mapper (lsst-scipipe) pprice@tiger2-sumire:/tigress/pprice/pipe2d-619 $ ingestPfsImages.py DATA --pfsConfigDir /projects/HSC/PFS/Subaru/drp/pfsDesign --config parse.pfsDesignId=1099528409104 --ignore-ingested -- <span class="code-quote">'/projects/HSC/PFS/Subaru/raw/2020-05-*/sps/PFSA*.fits'</span> <span class="code-quote">'/projects/HSC/PFS/Subaru/raw/2020-06-*/sps/PFSA*.fits'</span> (lsst-scipipe) pprice@tiger2-sumire:/tigress/pprice/pipe2d-619 $ mkdir CALIB (lsst-scipipe) pprice@tiger2-sumire:/tigress/pprice/pipe2d-619 $ ingestPfsCalibs.py DATA --calib CALIB --validity=1800 --doraise --mode=copy ~/pfs/drp_pfs_data/detectorMap/detectorMap-2019Jul-*.fits (lsst-scipipe) pprice@tiger2-sumire:/tigress/pprice/pipe2d-619 $ generateCommands.py DATA --calib CALIB --rerun calibs --blocks=pipe2d_618 -j 20 ../pipe2d-618/pipe2d-618.yaml calibs.sh generateCommands INFO: Start writing shell commands on <span class="code-quote">'calibs.sh'</span> generateCommands INFO: Processing calib block <span class="code-quote">'pipe2d_618'</span> generateCommands INFO: End writing shell commands on <span class="code-quote">'calibs.sh'</span> (lsst-scipipe) pprice@tiger2-sumire:/tigress/pprice/pipe2d-619 $ bash calibs.sh constructFiberTrace.py DATA --calib=CALIB --rerun=fiberTrace --doraise --batch-type=smp --cores=10 --id visit=18228..18231 constructFiberTrace.py DATA --calib=CALIB --rerun=fiberTrace --doraise --batch-type=smp --cores=10 --id visit=18236..18238 constructFiberTrace.py DATA --calib=CALIB --rerun=fiberTrace --doraise --batch-type=smp --cores=10 --id visit=18245..18248 ... (lsst-scipipe) pprice@tiger2-sumire:/tigress/pprice/pipe2d-619 $ ingestPfsCalibs.py DATA --calib CALIB --validity 1800 --doraise --mode=copy DATA/rerun/fiberTrace/FIBERTRACE/pfsFiberTrace-*.fits &gt;&gt;&gt; from lsst.daf.persistence <span class="code-keyword">import</span> Butler &gt;&gt;&gt; butler = Butler(<span class="code-quote">"DATA"</span>, calibRoot=<span class="code-quote">"/tigress/pprice/pipe2d-619/CALIB-test"</span>) &gt;&gt;&gt; butler.get(<span class="code-quote">"fiberTrace_filename"</span>, visit=18228, arm=<span class="code-quote">"r"</span>)[0] <span class="code-quote">'/tigress/pprice/pipe2d-619/CALIB-test/FIBERTRACE/pfsFiberTrace-2020-06-02-018228-r1.fits'</span> &gt;&gt;&gt; butler.get(<span class="code-quote">"fiberTrace_filename"</span>, visit=18236, arm=<span class="code-quote">"r"</span>)[0] <span class="code-quote">'/tigress/pprice/pipe2d-619/CALIB-test/FIBERTRACE/pfsFiberTrace-2020-06-02-018236-r1.fits'</span> </pre> </div></div> <p>Merged to master.</p> Development Rank 0|02qps2:ki Reviewers hassan Sprint 2DDRP-2021 A Story Points 3.0