https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1088 DRP 2-D Pipeline <p>With <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-668" title="Support ISR processing for NIR data" class="issue-link" data-issue-key="PIPE2D-668"><del>PIPE2D-668</del></a>, <tt>n</tt>-band data need to be compatible with the PSFB format. This is not the case for the current weekly <tt>n</tt> data, so a temporary fix was added to the <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-668" title="Support ISR processing for NIR data" class="issue-link" data-issue-key="PIPE2D-668"><del>PIPE2D-668</del></a> ticket branch <tt>pfs_pipe2d</tt> repo, such that <tt>n</tt> data are ignored.</p> <p>Please introduce PFSB <tt>n</tt> band data to the weekly build, and remove the temporary fix to the <tt>pfs_pipe2d</tt> repo.</p> PIPE2D-1088

Add PFSB-compliant NIR data to weeklies

Task Normal Done Done hassan hassan Thu, 13 Oct 2022 16:40:55 +0000 Tue, 6 Dec 2022 23:37:10 +0000 Tue, 6 Dec 2022 23:37:10 +0000 0 6 <p>This work appears to be blocked by <a href="https://jira.lsstcorp.org/browse/DM-35718" class="external-link" rel="nofollow">DM-35718</a> .</p> <p>Blockage by DM-35718 (<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1102" title="Import DM-35718 solution for Gen2 datasetExists of bypass data" class="issue-link" data-issue-key="PIPE2D-1102"><del>PIPE2D-1102</del></a>) no longer the case. However, as bias, dark and flat corrections need to be applied to <tt>b+r</tt> processing, but not for <tt>n</tt>, for the science processing two separate runs need to be made with <tt>reduceExposure.py</tt>: one for b+r using bias, dark and flat corrections, and one for `n` without bias, dark and flat corrections. This specific feature is currently not supported in the YAML command generation mechanism. <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=sogo.mineo" class="user-hover" rel="sogo.mineo">sogo.mineo</a> has been informed and looking into a means of supporting this.</p> <p>We should simply disable the loading of bias and dark for <tt>n</tt> in our custom <tt>IsrTask</tt>. I think we want to keep the flats, though.</p> <p>Disabling the bias and dark for <tt>n</tt> in <tt>IsrTask</tt> would be good. But just want to re-iterate (mainly for Mineo-san's information) that if the flats need to be configured separately for <tt>n</tt>, an update to the command generation mechanism is still needed.</p> <p>I began working on extending the command generation mechanism.</p> <p>Where is the I/O done? They are ignored in the processing.</p> <p>Can I add a commit to the current <tt>tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1088" title="Add PFSB-compliant NIR data to weeklies" class="issue-link" data-issue-key="PIPE2D-1088"><del>PIPE2D-1088</del></a></tt> branch worked on by Hassan? Is it legitimate act?</p> <p>This is the commit I want to push:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> generateCommands.py: Introduce subgroups of ScienceStep After <span class="code-keyword">this</span> patch is applied, the following construction (`group:...`) will be valid. scienceBlock: - name: process_brn id: [<span class="code-quote">"visit=47"</span>, <span class="code-quote">"arm=b^r^n"</span>] policy: reduceExposure: config: [<span class="code-quote">"isr.doDark=False"</span>] group: - id: [<span class="code-quote">"arm=b^r"</span>] - id: [<span class="code-quote">"arm=n"</span>] config: [<span class="code-quote">"isr.doBias=False"</span>] In <span class="code-keyword">this</span> <span class="code-keyword">case</span>, <span class="code-quote">"arm=b^r"</span> will be processed with `--config isr.doDark=False` and <span class="code-quote">"arm=n"</span> will be processed with `--config isr.doDark=False isr.doBias=False`. </pre> </div></div> <p>Sure <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=sogo.mineo" class="user-hover" rel="sogo.mineo">sogo.mineo</a> feel free to add commits to <tt>tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1088" title="Add PFSB-compliant NIR data to weeklies" class="issue-link" data-issue-key="PIPE2D-1088"><del>PIPE2D-1088</del></a></tt> as you wish.</p> <p>We shouldn't need to change anything in the command generation. We just need to tell the ISR that we don't need bias and dark for <tt>n</tt>. The flats will be required for <tt>n</tt>, just as for <tt>b</tt> and <tt>r</tt>.</p> <p>I've changed the <tt>PfsIsrTask</tt> so it won't attempt to load the bias or dark for <tt>arm=n</tt>. This should allow us to process CCD and H4RG data with the same configuration.</p> <p>OK. I'll incorporate your changes Paul in my changes to the weekly build, and get back to you if there's any issues.</p> <p>This may no longer be necessary, but here is the branch I made yesterday (Hassan told me about personal branches): <tt>u/SogoMineo/20221109a</tt></p> <p>Converted n-band PFFA files (generated by 2D simulator) to PFFB format. ISR-corrected images were first generated, and then converted to PFFB format using script <tt>dev_2ddrp/scripts/convertPFSAtoPFSB.py</tt>. For details of conversion process, see notes in <a href="https://github.com/Subaru-PFS/dev_2ddrp/blob/366f2bad38321de16cafb3197163a2bf30f7f013/notes/pipe2d-1088-notes.md" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/dev_2ddrp/blob/366f2bad38321de16cafb3197163a2bf30f7f013/notes/pipe2d-1088-notes.md</a> .</p> <p>Code changes look reasonable; I've not looked at the data.</p> <p>Merged to master</p> Blocks PIPE2D-1102 Relates PIPE2D-1095 PIPE2D-668 Development Rank 0|02qpjd:00r3000k Reviewers price Story Points 2.0