https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-863 DRP 2-D Pipeline <p>We have found that the expected illuminated fibers from bundle yellow and blue were incorrect. Now that the dummyCableB design has been fixed, we need to re-generate pfsConfig file and re-ingest the large dcb-21-fibers dataset (visit=51400..63009)</p> PIPE2D-863

re-generate pfsConfig for blue and yellow bundles, ingest data again

Task Normal Done Done arnaud.lefur arnaud.lefur Mon, 28 Jun 2021 11:13:48 +0000 Mon, 28 Jun 2021 19:31:24 +0000 Mon, 28 Jun 2021 19:31:24 +0000 0 2 <p>the 21-fibers and 23-fibers design files has been re-generated.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(lsst-scipipe) [afur@tiger2-sumire w.2021.26]$ makeDummyCableBDesign.py --directory /projects/HSC/PFS/Subaru/pfsConfig orange blue yellow WARNING: VerifyWarning: Card is too long, comment will be truncated. [astropy.io.fits.card] Wrote pfsDesign-0x0000100000000101.fits (lsst-scipipe) [afur@tiger2-sumire w.2021.26]$ makeDummyCableBDesign.py --directory /projects/HSC/PFS/Subaru/pfsConfig orange blue yellow red2 red7 WARNING: VerifyWarning: Card is too long, comment will be truncated. [astropy.io.fits.card] Wrote pfsDesign-0x0000101000010101.fits </pre> </div></div> <p>I tried to re-ingest the data, but fails with <em>already ingested</em></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(lsst-scipipe) [afur@tiger2-sumire ~]$ ingestPfsImages.py /tigress/HSC/PFS/Subaru/ --pfsConfigDir=/tigress/HSC/PFS/Subaru/pfsConfig --mode=link /tigress/HSC/PFS/Subaru/raw/2021-05-12/sps/PFSA051* --config clobber=True CameraMapper INFO: Loading exposure registry from /tigress/HSC/PFS/Subaru/registry.sqlite3 ingestPfs WARN: /tigress/HSC/PFS/Subaru/raw/2021-05-12/sps/PFSA05139411.fits: already ingested: {'site': 'S', 'category': 'A', 'visit': 51394, 'filter': 'b', 'arm': 'b', 'spectrograph': 1, 'ccd': 0, 'expTime': 8.0, 'taiObs': '2021-05-12T20:51:36.570', 'field': 'SEEING_TEST', 'dataType': 'COMPARISON', 'pfsDesignId': 1099528409104, 'slitOffset': 0.0, 'lamps': 'Ar', 'dateObs': '2021-05-12', 'dither': -0.0, 'shift': -1e-05, 'focus': 0.0, 'attenuator': -9998.0, 'photodiode': -9998.0} ingestPfs INFO: /tigress/HSC/PFS/Subaru/raw/2021-05-12/sps/PFSA05139411.fits --&lt;link&gt;--&gt; /tigress/HSC/PFS/Subaru/2021-05-12/PFSA051394b1.fits Traceback (most recent call last): File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/obs_pfs/w.2021.26/bin/ingestPfsImages.py", line 3, in &lt;module&gt; PfsIngestTask.parseAndRun() File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/18.1.0/python/lsst/pipe/tasks/ingest.py", line 416, in parseAndRun task.run(args) File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/18.1.0/python/lsst/pipe/tasks/ingest.py", line 553, in run self.register.addRow(registry, info, dryrun=args.dryrun, create=args.create) File "/tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/obs_pfs/w.2021.26/python/lsst/obs/pfs/ingest.py", line 439, in addRow conn.cursor().execute(sql, values) sqlite3.IntegrityError: UNIQUE constraint failed: raw.site, raw.category, raw.visit, raw.filter, raw.arm, raw.spectrograph, raw.pfsDesignId </pre> </div></div> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=price" class="user-hover" rel="price">price</a> how do you usually proceed ?</p> <p>Are you changing the <tt>pfsDesignId</tt>? If so, we'd need to delete the ingested rows by hand. Otherwise, <tt>--config clobber=True register.ignore=True</tt> is what you want.</p> <p>the pfsDesignId value is the same but the interpretation from bundle name to fiberId is different.<br/> So I went ahead and tried : </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ingestPfsImages.py /tigress/HSC/PFS/Subaru/ --pfsConfigDir=/tigress/HSC/PFS/Subaru /pfsConfig --mode=link /tigress/HSC/PFS/Subaru/raw/2021-05-13/sps/PFSA0514371* --config clobber=True register.ignore=True CameraMapper INFO: Loading exposure registry from /tigress/HSC/PFS/Subaru/registry.sqlite3 ingestPfs INFO: /tigress/HSC/PFS/Subaru/raw/2021-05-13/sps/PFSA05143711.fits --&lt;link&gt;--&gt; /tigress/HSC/PFS/Subaru/2021-05-13/PFSA051437b1.fits ingestPfs INFO: /tigress/HSC/PFS/Subaru/raw/2021-05-13/sps/PFSA05143712.fits --&lt;link&gt;--&gt; /tigress/HSC/PFS/Subaru/2021-05-13/PFSA051437r1.fits </pre> </div></div> <p>But the pfsConfig from the butler still looks wrong :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>repoRoot = "/tigress/HSC/PFS/Subaru" butler = dafPersist.Butler(repoRoot) pfsConfig = butler.get('pfsConfig', dataId, visit=51437) pfsConfig.fiberId[pfsConfig.fiberId&lt;652] array([ 12, 32, 60, 110, 111, 161, 210, 223, 259, 289, 341, 347, 400, 418, 449, 466, 518, 545, 593, 620, 641], dtype=int32) </pre> </div></div> <p>I'm guessing there's no facility to clobber the <tt>pfsConfig</tt>. That's something I could add, but it's probably not worth it given that we're going to switch the middleware soon. Can you delete the generated <tt>pfsConfig</tt> files (the originals, not the links) by hand?</p> <p>okay just did it.<br/> I had to do it twice because I badly specified the --pfsConfigDir in the first place.<br/> --pfsConfigDir=/tigress/HSC/PFS/Subaru/pfsConfig instead of --pfsConfigDir=/tigress/HSC/PFS/Subaru/drp/pfsDesign</p> <p>fiberIds should now be correctly identified.</p> Development Rank 0|02qpt6:508yl0chzzy0000026m Sprint 2DDRP-2021 A 6