https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1301 DRP 2-D Pipeline <p><tt>generateCommands.py</tt> currently uses <tt>constructFiberProfiles.py</tt> to build fiber profiles. <tt>reduceProfiles.py</tt> is an alternative fiber profile construction code that fits profiles to multiple images simultaneously, and is preferred for use on Subaru data because some cobras cannot be hidden. Please support <tt>reduceProfiles</tt> in <tt>generateCommands</tt>.</p> PIPE2D-1301

add support for reduceProfiles in generateCommands

Story Normal Done Done sogo.mineo price Tue, 19 Sep 2023 21:04:05 +0000 Fri, 29 Sep 2023 18:42:40 +0000 Fri, 29 Sep 2023 18:42:40 +0000 0 2 <p>Does it suffice to cease using <tt>constructFiberProfiles.py</tt> and to use <tt>reduceProfiles.py</tt> instead, or should <tt>constructFiberProfiles.py</tt> be kept as an option?</p> <p><tt>reduceProfiles.py</tt> does not take "<tt>&#45;&#45;batch-type=smp &#45;&#45;cores=$n</tt>". Is it possible to modify <tt>reduceProfiles.py</tt> so that it will take <tt>&#45;&#45;batch-type</tt> option? Should I make a special override for <tt>reduceProfiles.py</tt>?</p> <p>I think we can replace <tt>constructFiberProfiles</tt> with <tt>reduceProfiles</tt>, but we'll need to test it on the integration test and weekly. One advantage to replacing it is that <tt>reduceProfiles</tt> writes a complete set of fiberProfiles: there's no need to run <tt>constructFiberProfiles</tt> twice and then merge the profiles.</p> <p><tt>reduceProfiles</tt> uses <tt>CmdLineTask</tt> rather than being based on the usual calib construction, so it doesn't support <tt>&#45;&#45;batch&#45;type</tt> and <tt>&#45;&#45;cores</tt>.</p> <p>Change of integration_test.yaml:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> --- a/examples/integration_test.yaml +++ b/examples/integration_test.yaml @@ -28,15 +28,7 @@ calibBlock: config: - <span class="code-quote">"isr.doCrosstalk=False"</span> # No crosstalk in simulated data fiberProfiles: - group: - - - id: <span class="code-quote">"field=FLAT_ODD"</span> - config: - - <span class="code-quote">"isr.doCrosstalk=False"</span> # No crosstalk in simulated data - - - id: <span class="code-quote">"field=FLAT_EVEN"</span> - config: - - <span class="code-quote">"isr.doCrosstalk=False"</span> # No crosstalk in simulated data + id: <span class="code-quote">"field=FLAT_ODD^FLAT_EVEN"</span> detectorMap: id: <span class="code-quote">"field=ARC"</span> config: </pre> </div></div> <p>Generated command line:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> reduceProfiles.py /data22a/mineo/pfswork/pfsrepos/pfs_pipe2d/TEST/INTEGRATION --calib=/data22a/mineo/pfswork/pfsrepos/pfs_pipe2d/TEST/INTEGRATION/CALIB --rerun=integration/test_calib/fiberProfiles --longlog=1 -j10 --doraise --id <span class="code-quote">'field=FLAT_ODD^FLAT_EVEN'</span> </pre> </div></div> <p>I get this error:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Traceback (most recent call last): File <span class="code-quote">"/data22a/mineo/pfswork/pfsrepos/drp_stella/bin/reduceProfiles.py"</span>, line 3, in &lt;module&gt; ReduceProfilesTask.parseAndRun() File <span class="code-quote">"/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py"</span>, line 688, in parseAndRun resultList = taskRunner.run(parsedCmd) File <span class="code-quote">"/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py"</span>, line 240, in run targetList = self.getTargetList(parsedCmd) File <span class="code-quote">"/data22a/mineo/pfswork/pfsrepos/drp_stella/python/pfs/drp/stella/reduceProfiles.py"</span>, line 44, in getTargetList raise RuntimeError(f<span class="code-quote">"No norm provided <span class="code-keyword">for</span> spectrograph={spectrograph} arm={arm}"</span>) RuntimeError: No norm provided <span class="code-keyword">for</span> spectrograph=1 arm=b </pre> </div></div> <p>Do I have to give "config=..." options to let reduceProfiles.py run with the integration test data?</p> <p>Oh, sorry. There's an additional <tt>&#45;&#45;normId</tt> argument, which provides a quartz at full fiber density to measure for the normalisation of the profiles. In the case of the integration test, it should probably be something like <tt>&#45;&#45;normId field=FLAT</tt>.</p> <p>reduceProfiles.py now runs, but does not end in success.</p> <p>Yaml file:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> --- a/examples/integration_test.yaml +++ b/examples/integration_test.yaml @@ -28,15 +28,8 @@ calibBlock: config: - <span class="code-quote">"isr.doCrosstalk=False"</span> # No crosstalk in simulated data fiberProfiles: - group: - - - id: <span class="code-quote">"field=FLAT_ODD"</span> - config: - - <span class="code-quote">"isr.doCrosstalk=False"</span> # No crosstalk in simulated data - - - id: <span class="code-quote">"field=FLAT_EVEN"</span> - config: - - <span class="code-quote">"isr.doCrosstalk=False"</span> # No crosstalk in simulated data + id: <span class="code-quote">"field=FLAT_ODD^FLAT_EVEN"</span> + normId: <span class="code-quote">"field=FLAT"</span> detectorMap: id: <span class="code-quote">"field=ARC"</span> config: </pre> </div></div> <p>Generated command line:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> reduceProfiles.py /data22a/mineo/pfswork/pfsrepos/pfs_pipe2d/TEST/INTEGRATION --calib=/data22a/mineo/pfswork/pfsrepos/pfs_pipe2d/TEST/INTEGRATION/CALIB --rerun=integration/test_calib/fiberProfiles --longlog=1 -j10 --doraise --id <span class="code-quote">'field=FLAT_ODD^FLAT_EVEN'</span> --normId field=FLAT </pre> </div></div> <p>Error message:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Traceback (most recent call last): File <span class="code-quote">"/data22a/mineo/pfswork/lsst_stack/lsst_home/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/multiprocessing/pool.py"</span>, line 125, in worker result = (True, func(*args, **kwds)) File <span class="code-quote">"/data22a/mineo/pfswork/lsst_stack/lsst_home/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/multiprocessing/pool.py"</span>, line 48, in mapstar <span class="code-keyword">return</span> list(map(*args)) File <span class="code-quote">"/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py"</span>, line 433, in __call__ result = self.runTask(task, dataRef, kwargs) File <span class="code-quote">"/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py"</span>, line 504, in runTask <span class="code-keyword">return</span> task.runDataRef(dataRef, **kwargs) File <span class="code-quote">"/data22a/mineo/pfswork/pfsrepos/drp_stella/python/pfs/drp/stella/reduceProfiles.py"</span>, line 190, in runDataRef normList = [self.processExposure(ref) <span class="code-keyword">for</span> ref in normRefList] File <span class="code-quote">"/data22a/mineo/pfswork/pfsrepos/drp_stella/python/pfs/drp/stella/reduceProfiles.py"</span>, line 190, in &lt;listcomp&gt; normList = [self.processExposure(ref) <span class="code-keyword">for</span> ref in normRefList] File <span class="code-quote">"/data22a/mineo/pfswork/pfsrepos/drp_stella/python/pfs/drp/stella/reduceProfiles.py"</span>, line 241, in processExposure <span class="code-keyword">return</span> self.reduceExposure.runDataRef(dataRef) File <span class="code-quote">"/data22a/mineo/pfswork/pfsrepos/drp_stella/python/pfs/drp/stella/reduceExposure.py"</span>, line 261, in runDataRef calibs = self.getSpectralCalibs(sensorRef, exposure, pfsConfig) File <span class="code-quote">"/data22a/mineo/pfswork/pfsrepos/drp_stella/python/pfs/drp/stella/reduceExposure.py"</span>, line 524, in getSpectralCalibs detectorMap = self.adjustDetectorMap.run( File <span class="code-quote">"/data22a/mineo/pfswork/pfsrepos/drp_stella/python/pfs/drp/stella/adjustDetectorMap.py"</span>, line 69, in run raise RuntimeError(f<span class="code-quote">"Insufficient good lines: {numGoodLines} vs {needNumLines}"</span>) RuntimeError: Insufficient good lines: 0 vs 24 """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File <span class="code-quote">"/data22a/mineo/pfswork/pfsrepos/drp_stella/bin/reduceProfiles.py"</span>, line 3, in &lt;module&gt; ReduceProfilesTask.parseAndRun() File <span class="code-quote">"/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py"</span>, line 688, in parseAndRun resultList = taskRunner.run(parsedCmd) File <span class="code-quote">"/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py"</span>, line 244, in run resultList = list(mapFunc(self, targetList)) File <span class="code-quote">"/data22a/mineo/pfswork/lsst_stack/lsst_home/stack/miniconda3-py38_4.9.2-3.0.0/Linux64/pipe_base/g590c34a36e+5da9528084/python/lsst/pipe/base/cmdLineTask.py"</span>, line 47, in _runPool <span class="code-keyword">return</span> pool.map_async(function, iterable).get(timeout) File <span class="code-quote">"/data22a/mineo/pfswork/lsst_stack/lsst_home/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-3.0.0/lib/python3.8/multiprocessing/pool.py"</span>, line 771, in get raise self._value RuntimeError: Insufficient good lines: 0 vs 24 </pre> </div></div> <p>I pushed the latest ticket branch. Do you have any suggestion?</p> <p>I found two problems. First, we need to use <tt>&#45;&#45;normId field=FLAT dither=0</tt>, in order to exclude the dithered flats. Second, I found some problems with reduceProfiles after an API change in reduceExposures earlier; I've put a fix on this ticket branch.</p> <p>Thanks. The integration test now runs successfully. I have made PRs in github for both <tt>pfs_pipe2d</tt> and <tt>drp_stella</tt>, but I am not sure whether I can ask you <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=price" class="user-hover" rel="price">price</a> to review them, for the commit to <tt>drp_stella</tt> was made by you. Who should I ask to review them?</p> <p>I tried running the weekly, since this is a big change to the workflow, and found a couple of problems. Firstly, the configuration overrides for <tt>reduceProfiles</tt> got stripped out, which causes the weekly to fail because it can't find the NIR IPC calibrations. I put the configuration overrides back on a new commit, but you're welcome to squash it with your commit if you like.<br/> The second problem is more fundamental: there aren't any <tt>arm=m field=FLAT</tt> exposures in the weekly to use for the <tt>&#45;&#45;normId</tt>. I'll need to generate some.</p> <p>Thank you. Now I think we can close this issue. Should I ask someone else to be the reviewer? (If so, whom should I ask?)</p> <p>Sorry, I'm still working on updating the weekly. I'm about to start it running before heading off to bed. I'll finish it up in the morning.<br/> I think it would be fine if you review my changes and I review yours.</p> <p>I finally got the weekly to pass. Would you please review the changes I made in pfs&#95;pipe2d and drp&#95;stella?</p> <p>Thank you for your work. I made a comment on <tt>drp_stella</tt>, but everything else looks good. Could you review the commit I first made?</p> <p>Your changes were great, thanks.</p> <p>Since everything is in order and the tests pass, I've gone ahead and merged to master.</p> <p>Thanks again!</p> Relates PIPE2D-1306 Development Rank 0|02qph7:06r2 Reviewers price