https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-506 DRP 2-D Pipeline <p>Following <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-442" title="Upgrade to LSST stack v18.1.0" class="issue-link" data-issue-key="PIPE2D-442"><del>PIPE2D-442</del></a>, the 2D DRP pipeline makes use of the LSST stack v18.1.0 that contains the generation-3 (Gen3) middleware for pipeline tasks and butler I/O. The gen2 middleware that is currently being used by the pipeline will be phased out around the end of 2020. To avoid potential disruptions during commissioning, migrate code to use the gen3 middleware.</p> PIPE2D-506

Migrate code to use LSST gen3 middleware

Story Normal Done Done price hassan Tue, 7 Jan 2020 14:46:33 +0000 Wed, 25 May 2022 19:11:41 +0000 Wed, 25 May 2022 19:11:41 +0000 0 3 <p>This ticket should be blocked on another ticket to upgrade the LSST base stack.</p> <p>Because a ready Gen3 middleware is not yet available, and Gen2 will be supported for at least another year, I suggest we needn't think about this ticket for another six months. The right time to think about this will be when LSST start using Gen3 for their fortnightly RC2 runs.</p> <p>Thought: the Gen3 butler doesn't care about filename templates. We're going to need a facility to extract files from the butler and write them with the correct filename template for distribution.</p> <p>I've got the entire calib and science pipelines working under Gen3. There is more work that needs to be done, but having this merged will allow us to work on the below tasks one by one:<br/> 1. Add scripts to export products from the Gen3 datastore.<br/> 2. Check quality of Gen3 reduction compared to Gen2.<br/> 3. Add Gen3 support to bootstrapDetectorMap and new flux calibration tasks from NAOJ.<br/> 4. Check that we can run real data with Gen3.<br/> 5. Write Gen3 PFS user guide.<br/> 6. Port weekly to Gen3.<br/> 7. Set up shared Gen3 datastore with Postgresql registry.<br/> 8. Regenerate Subaru calibs with Gen3.<br/> 9. Investigate bash completion for "butler" and "pipetask" commands.<br/> 10. Investigate parsl plugin for processing on cluster.</p> <p>This effectively replaces <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-598" title="Upgrade base LSST stack to latest version" class="issue-link" data-issue-key="PIPE2D-598"><del>PIPE2D-598</del></a>: there's little point in merging that one if we're going to merge this one soon after, as it requires another change to the base LSST stack, from 23.0.0 to <tt>w&#95;2022&#95;17</tt>.</p> <p>Reviewed pull requests related to this ticket. Changes look fine. </p> <p>Also looked at the <tt>pfsMerged</tt> output from an example SuNSS visit (71280) processed with the <tt>tickets\<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-506" title="Migrate code to use LSST gen3 middleware" class="issue-link" data-issue-key="PIPE2D-506"><del>PIPE2D-506</del></a></tt> branches and with a recent weekly <tt>w.2022.18</tt>. A 'spectral image' (generated using the <tt>showAllSpectraAsImage</tt> utility but setting the flux as the difference of the flux values between the two processing runs is shown in figures <span class="image-wrap" style=""><a id="15180_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15180/15180_Screen+Shot+2022-05-06+at+3.28.15+PM.png" title="Screen Shot 2022-05-06 at 3.28.15 PM.png" file-preview-type="image" file-preview-id="15180" file-preview-title="Screen Shot 2022-05-06 at 3.28.15 PM.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15181/_thumb_15181.png" style="border: 0px solid black" /></a></span> and <span class="image-wrap" style=""><a id="15180_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/15180/15180_Screen+Shot+2022-05-06+at+3.28.15+PM.png" title="Screen Shot 2022-05-06 at 3.28.15 PM.png" file-preview-type="image" file-preview-id="15180" file-preview-title="Screen Shot 2022-05-06 at 3.28.15 PM.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/15180/_thumb_15180.png" style="border: 0px solid black" /></a></span> In general the difference is close to zero, with the difference at the blue end being noiser. A strange feature around 502nm for fiberIds around 338 is seen. As suggested by Paul, and agreed, this should not stop the merging of this ticket. Future work on such anomalies can be addressed as part of <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1031" title="Check quality of Gen3 reduction compared to Gen2" class="issue-link" data-issue-key="PIPE2D-1031">PIPE2D-1031</a>.</p> <p>I'd have expected the results to be equal to close to machine precision. This needs to be investigated, but doing so as part of <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1031" title="Check quality of Gen3 reduction compared to Gen2" class="issue-link" data-issue-key="PIPE2D-1031">PIPE2D-1031</a> is fine.</p> <p>My first guess is that there are configuration parameters that differ between the pipelines. This is exactly what <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1031" title="Check quality of Gen3 reduction compared to Gen2" class="issue-link" data-issue-key="PIPE2D-1031">PIPE2D-1031</a> was created for. We will not drop Gen2 support until we are convinced the Gen3 pipeline is performing at the level of Gen2.</p> <p>Merging this work soon is important, since there are a bunch of changes that I don't want to conflict with other work. On the other hand, we don't want to merge such a large change right before the observing run on the off chance that it causes problems. I propose to merge immediately after the observing run, and in the mean time base other non-urgent work off of this ticket branch.</p> <p>Merged.</p> Blocks PIPE2D-597 PIPE2D-598 Relates PIPE2D-1031 Development Rank 0|02qpjd:040z Reviewers hassan Sprint 2DDRP-2022 B 2DDRP-2022 C 2DDRP-2022 D Story Points 4.0