https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-761 DRP 2-D Pipeline <p>reduceExposure currently appears to process all the data requested before writing any of it out, processing all the e.g. b data before starting on r.</p> <p>This blows up its memory footprint and frustrates anyone waiting for a given visit to finish.  </p> <p>Please restructure things to process all the arms for a given visit, write out the results, and clean up memory before proceeding to the next visit.</p> PIPE2D-761

Restructure reduceExposure to process visit one-by-one

Story Normal Done Done price rhl Sat, 6 Mar 2021 20:33:02 +0000 Thu, 11 Mar 2021 21:07:13 +0000 Thu, 11 Mar 2021 21:07:13 +0000 0 3 <p><tt>reduceExposure</tt> acts like <tt>processCcd</tt>: it most certainly writes out the results as you go along. As for the memory footprint, I'm not sure, but I don't think there's any reason why it should be different than <tt>processCcd</tt>.</p> <p>I don't think so.  It processes all the data to form <tt>exposureList</tt> before starting any extractions, then loops over all the exposures, and then finally calls <tt>self.write</tt>.  Please check.</p> <p>I'm sorry, you're right that it works through an <tt>exposureList</tt>. However, I believe that <tt>exposureList</tt> consists only of exposures from the same visit (i.e., all arms from all spectrographs), so we truly do <a href="https://github.com/Subaru-PFS/drp_stella/blob/master/python/pfs/drp/stella/reduceExposure.py#L69-L77" class="external-link" rel="nofollow">parallelise over visits</a>. The reason for this is that we want to derive a single 2D sky subtraction solution from all exposures in the same visit, and that has to happen before we extract the spectra.</p> <p>I propose that we should create a new <tt>CmdLineTask</tt> to perform a single-arm spectra extraction, which will allow faster processing of SuNSS data.</p> <p>I thought I'd commented on this already...</p> <p>I think we want to pass in a list of visits and do the right thing (process all 12 arms for each visit before proceeding to the next visit), and each visit is independent (and may have its own pfsConfig).</p> <p>I don't think we need a new command;  if I process a single spectrograph and two bands it'll process all of that visit before moving on.</p> <p>When you say, "we want to pass in a list of visits", are you referring to using the <tt>ReduceExposureTask</tt> directly in python, and therefore working around the <tt>ReduceExposureRunner</tt> that ensures we operate on a visit at a time?</p> <p>I'm talking about the command line.  If I run</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> reduceExposure.py /projects/HSC/PFS/Subaru --id visit=50000..500100 arm=b^r</pre> </div></div> <p>logging suggests that it runs ISR (etc.) and extracts spectra and writes pfsArm files from all the <tt>b</tt> visits in turn before starting on the <tt>r</tt> ones. We only have one spectrograph, but from what you say it'd have handled all 4 spectrographs in <tt>b</tt> before doing the extraction. I'd like to process all the <tt>b</tt> <em>and</em> all the <tt>r</tt> before moving on to the next visit, but as I can achieve this (more or less) by running two jobs, one for b and one for r, this isn't all that urgent. If we add an option (or a new command) to also run the merge step this will become important again.</p> <p>I am still worried about when it releases the memory, as it seemed to crash on very long visit lists, but that's a different issue. Maybe this was just a problem on the mountain on small memory machines (I just processed 632 b^r visits on tiger without problems).</p> <p>So the <tt>ReduceExposureTask</tt> is fine and doesn't need to be restructured (it operates on all arms of the same kind in an exposure, and that's the way it needs to operate for sky subtraction), but the request is to change the order of the inputs that it receives, which is set by the <tt>ReduceExposureRunner</tt>. I've made a small change which reorders the inputs so that it iterates over visits more quickly than arms.</p> <p>Note that I'm not sure how python's <tt>multiprocessing</tt> (used when we specify a <tt>-j</tt> flag) will change the ordering we provide, but hopefully it will do something approaching FIFO.</p> <p>Approved with no additional changes requested.</p> <p>Merged to master.</p> Development Rank 0|zzs47y:zzzzw Reviewers hassan Sprint 2DDRP-2021 A3