https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-914 DRP 2-D Pipeline <p><a href="https://pfspipe.ipmu.jp/jira/browse/INSTRM-1397" title="Fast SpS reads for cobra moves behind blackspots" class="issue-link" data-issue-key="INSTRM-1397"><del>INSTRM-1397</del></a> request a new format for fast SpS exposures to aid in cobra positioning. Once that is available, please process that in the DRP (ISR and flux measurement per fiber).</p> PIPE2D-914

Add support for SpS fast-mode exposure format

Task Normal Done Done price hassan EngRun Mon, 11 Oct 2021 17:38:01 +0000 Fri, 12 Nov 2021 00:48:58 +0000 Fri, 12 Nov 2021 00:48:58 +0000 0 4 <p>How are the results to be delivered?</p> <p>For row-windowed reads, I propose normal-sized PFSA files with the unread rows set to 0, and with some informative keywords. Or would you prefer something far different from regular PFSA files? Not sure what I'd do about binned images.</p> <p>For row-windowed reads, I like the solution of setting unread rows to zero: with compression enabled we won't be wasting space, and this enables us to use the usual code without much in the way of changes.</p> <p>I think binning in the spatial dimension should be ruled out, as any binning will confuse the fibers. I wonder what we could gain from binning in the spectral dimension that we can't achieve through windowing. The only thing I can think of is spectral coverage, but I can't imagine using arcs for this purpose rather than a continuum source. I therefore suggest we not consider any form of binning.</p> <p>But I forgot to say: my original question was regarding the format for the results from the pipeline (presumably a <tt>dict</tt> mapping fiberIds to flux measurements). Who/what needs the results, and what format would be most suitable?</p> <p>Jim-sensei believes that we'll need to bin. I'm not sure that he's right in this case, and am happy to postpone that part of the work.</p> <p>Why is this still in the sprint? I think we decided not to implement binning</p> <p>Agreed. Removing that ticket from the eng run sprint.</p> <p>When reading a band of rows, W_CDROW0 and W_CDROWN are added to the PHDU. All pixels outside that band are set to 0. Other that they, they are normal PFSA files. Will try getting some from SuNSS.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a> will provide <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=price" class="user-hover" rel="price">price</a> test data by tomorrow (Thu) evening for Paul to check.</p> <p>Example file ingested on tiger. visit=69294 arm=r. SuNSS within the dome; am guessing with dome lights.</p> <p><tt>reduceExposure.py</tt> kind of worked on the test data, but there were a few problems:<br/> 1. The speed was much too slow (80 sec, even with <tt>doAdjustDetectorMap=False</tt>).<br/> 2. The unread area was not masked.<br/> 3. The good area was on a pedestal of about 1400 counts.</p> <p>To fix:<br/> 1. Added a switch to <tt>reduceExposure</tt> to disable emission line photometry (<tt>doMeasureLines=False</tt>), which reduces the runtime down to about 14 sec.<br/> 2. Added code in ISR to mask zero pixels. This is done before overscan subtraction, so the bad pixels can be excluded from the overscan. Slightly annoyingly, this requires masking the pixels <tt>SAT</tt> (mask value for overscans is hard-wired). I added <tt>NO&#95;DATA</tt> to make it a bit clearer what's going on.<br/> 3. The great reduction in overscan real estate means we need to use a different overscan method, like <tt>overscanFitType=MEDIAN</tt>.</p> <p>I recommend a command-line something like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/ --rerun price/pipe2d-914 --id visit=69294 -c isr.doFlat=False doAdjustDetectorMap=False doMeasureLines=False isr.overscanFitType=MEDIAN </pre> </div></div> <p>We know that these data are going to need these tweaks. Can we look at the metadata and set them automatically?</p> <p>The detectorMap adjustment can be done (at least, in the red; wouldn't try it in the blue) at the cost of about 20 sec (total runtime 36 sec):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/ --rerun price/pipe2d-914 --id visit=69294 -c doWriteCalexp=True isr.doFlat=False doAdjustDetectorMap=True doMeasureLines=False isr.overscanFitType=MEDIAN centroidLines.doSubtractContinuum=False </pre> </div></div> <p>I don't know how we could possibly apply the adjustment by looking at the header.</p> <p>I meant adjustment to the configs. So that a simple <tt>reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/ --rerun price/pipe2d-914 --id visit=69294</tt> would work</p> <p>We could have the detectorMap adjustment and line measurement also respond to the presence of header keywords, but <tt>isr.overscanFitType</tt> is used deep in a massive LSST function, so overriding that would be real messy. We could do it, but blrgh!</p> <p>Sorry to add another request, but after reviewing the code, I would have thought it better to apply masks based on the (inverse to the) region defined by the <tt>W_CDROW0</tt> and <tt>W_CDROWN</tt> keywords. See earlier comment by <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a>.</p> <p>I think that the approach I've adopted (masking zero-valued pixels in the raw) is superior. It means I don't have to muck around with xy0 on the raw, and it catches problems in the full readouts as well.</p> <p>Back to configs. So the problem is that the configs are frozen before we read the metadata? That's true. So I think I'd add an option for "this sort of data" and have that set your magic values. Would that work?</p> <p>Why don't we just add a configuration file to be used for "this sort of data"?</p> <p>a `-C` file in a well-known place? Is that really cleaner?</p> <p>I don't know how to make it work cleanly otherwise.</p> <p>Can't you set values in the config validate function?</p> <p>We could tweak the ISR configuration based on a <tt>reduceExposure</tt> configuration, but not a header keyword.</p> <p>That was what my 'an option for "this sort of data" and have that set your magic values' proposal was supposed to mean.</p> <p>OK, here 'tis:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> reduceExposure.py /projects/HSC/PFS/Subaru --calib /projects/HSC/PFS/Subaru/CALIB-PFI-20211108/ --rerun price/pipe2d-914 --id visit=69294 -c isr.doFlat=False quickLook=True </pre> </div></div> <p>I'm sorry, but <tt>quicklook</tt> isn't correct &#8211; that implies e.g. cheap sky subtraction. This is closer to <tt>windowed</tt></p> <p>Renamed to <tt>windowed</tt>.</p> <p>Merged.</p> Blocks INSTRM-1414 INSTRM-1397 Relates INSTRM-1439 Development Rank 0|zzs9ul:xi Reviewers hassan Sprint 2DDRP-2021 A11 Story Points 4.0