PFS-JIRA https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 [PIPE2D-523] Demonstrate R, B and NIR arm merging https://pfspipe.ipmu.jp/jira/browse/PIPE2D-523 DRP 2-D Pipeline <p>Following the implementation of <a href="https://pfspipe.ipmu.jp/jira/browse/SIM2D-123" title="Model the NIR based on a CCD" class="issue-link" data-issue-key="SIM2D-123"><del>SIM2D-123</del></a> where a CCD-based NIR simulation is now provided, check that all 3 simulated arms can be merged successfully with the 2D DRP.</p> PIPE2D-523 Demonstrate R, B and NIR arm merging Story Normal Done Done price hassan Fri, 13 Mar 2020 22:25:45 +0000 Tue, 15 Nov 2022 15:52:20 +0000 Fri, 8 May 2020 19:40:15 +0000 0 4 <p>May as well include M also, since we now have sims working for that.</p> <p>It would be helpful to make a larger dataset than the integration test (which is size-limited by the Travis runtime limit).</p> <p>The work accomplished on this ticket is a bit more extensive than what's in the description above, but it's all related.</p> <ul> <li>Created a new test dataset that lives in <tt>/projects/HSC/PFS/weekly</tt>; the script to create it lives in pfs_pipe2d. The test dataset includes BRMN data, and will be used to demonstrate BRN and BMN merging.</li> <li>Put together a script for processing this data, and testing the results.</li> <li>Set up Jenkins so that it will run this script weekly (this may need some tweaking, but we're pretty close).</li> <li>Fixed some bugs and made some improvements after poking at the results of processing.</li> <li>Generated a new integration test dataset with 10 fibers instead of 600, so the integration test will go faster (full fiber density can be tested with the weekly).</li> </ul> <p>Do you have plans to use Mineo-kun's scripting system for running the code, replacing your script?</p> <p>Once Mineo-san's system is working, it would be good to see it replace both the integration test script and the weekly script.</p> <p>I have processed arc data taken at LAM on 2019 July with this branch. It seems to work fine up to reduceArc.py. Should I check mergeArms.py as well? In that case, what is the command line for that?</p> <p>This would be useful, but only if you have 1 hour spare. The command line is </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> mergeArms.py /path/to/dataRepo –calib /path/to/calibRepo –rerun pipeline –id field=OBJECT </pre> </div></div> <p>Note before running the above, you need to run <tt>reduceExposure</tt> first to create the <tt>pfsArm</tt> files:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> reduceExposure.py /path/to/dataRepo --calib /path/to/calibRepo --rerun pipeline --id field=OBJECT </pre> </div></div> <p><tt>mergeArms.py</tt> will fail with the following error. Any suggestion?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> (lsst-scipipe-1172c30) [yasuda@fe PIPE2D-523]$ mergeArms.py $REPO_DIR --calib $REPO_DIR/CALIB --rerun pipeline --id visit=21400 CameraMapper INFO: Loading exposure registry from /gpfs02/work/yasuda/PFS/calibs_Jul_PIPE2D-523/registry.sqlite3 CameraMapper INFO: Loading calib registry from /gpfs02/work/yasuda/PFS/calibs_Jul_PIPE2D-523/CALIB/calibRegistry.sqlite3 CameraMapper INFO: Loading calib registry from /gpfs02/work/yasuda/PFS/calibs_Jul_PIPE2D-523/CALIB/calibRegistry.sqlite3 root INFO: Running: /gpfs02/work/yasuda/PFS/work/PIPE2D-523/drp_stella/bin/mergeArms.py /gpfs02/work/yasuda/PFS/calibs_Jul_PIPE2D-523 --calib /gpfs02/work/yasuda/PFS/calibs_Jul_PIPE2D-523/CALIB --rerun pipeline --id visit=21400 WARNING: You are using OpenBLAS with multiple threads (20), but have not specified the number of threads using one of the OpenBLAS environment variables: OPENBLAS_NUM_THREADS, GOTO_NUM_THREADS, OMP_NUM_THREADS. This may indicate that you are unintentionally using multiple threads, which may cause problems. WE HAVE THEREFORE DISABLED OpenBLAS THREADING. If you know what you are doing and want threads enabled implicitly, set the environment variable LSST_ALLOW_IMPLICIT_THREADS. Traceback (most recent call last): File <span class="code-quote">"/opt/local/pfs2.1/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py"</span>, line 388, in __call__ result = self.runTask(task, dataRef, kwargs) File <span class="code-quote">"/opt/local/pfs2.1/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py"</span>, line 447, in runTask <span class="code-keyword">return</span> task.runDataRef(dataRef, **kwargs) File <span class="code-quote">"/gpfs02/work/yasuda/PFS/work/PIPE2D-523/drp_stella/python/pfs/drp/stella/mergeArms.py"</span>, line 96, in runDataRef sky1d = self.subtractSky1d.run(sum(spectra, []), pfsConfig, sum(lsf, [])) File <span class="code-quote">"/gpfs02/work/yasuda/PFS/work/PIPE2D-523/drp_stella/python/pfs/drp/stella/subtractSky1d.py"</span>, line 56, in run resampledList = self.resampleSpectra(spectraList, pfsConfig) File <span class="code-quote">"/gpfs02/work/yasuda/PFS/work/PIPE2D-523/drp_stella/python/pfs/drp/stella/subtractSky1d.py"</span>, line 84, in resampleSpectra <span class="code-keyword">return</span> [spectra.resample(wavelength, pfsConfig.fiberId[index]) <span class="code-keyword">for</span> spectra in spectraList] File <span class="code-quote">"/gpfs02/work/yasuda/PFS/work/PIPE2D-523/drp_stella/python/pfs/drp/stella/subtractSky1d.py"</span>, line 84, in &lt;listcomp&gt; <span class="code-keyword">return</span> [spectra.resample(wavelength, pfsConfig.fiberId[index]) <span class="code-keyword">for</span> spectra in spectraList] File <span class="code-quote">"/gpfs02/work/yasuda/PFS/work/PIPE2D-523/drp_stella/python/pfs/drp/stella/datamodel/pfsFiberArraySet.py"</span>, line 110, in resample <span class="code-keyword">return</span> type(self)(self.identity, fiberId, np.concatenate([[wavelength]]*numSpectra), ValueError: need at least one array to concatenate During handling of the above exception, another exception occurred: Traceback (most recent call last): File <span class="code-quote">"/gpfs02/work/yasuda/PFS/work/PIPE2D-523/drp_stella/bin/mergeArms.py"</span>, line 3, in &lt;module&gt; MergeArmsTask.parseAndRun() File <span class="code-quote">"/opt/local/pfs2.1/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py"</span>, line 603, in parseAndRun resultList = taskRunner.run(parsedCmd) File <span class="code-quote">"/opt/local/pfs2.1/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py"</span>, line 221, in run resultList = list(mapFunc(self, targetList)) File <span class="code-quote">"/opt/local/pfs2.1/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py"</span>, line 400, in __call__ <span class="code-quote">", "</span>.join(str(ref.dataId) <span class="code-keyword">for</span> ref in dataRef), eName, e) File <span class="code-quote">"/opt/local/pfs2.1/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py"</span>, line 400, in &lt;genexpr&gt; <span class="code-quote">", "</span>.join(str(ref.dataId) <span class="code-keyword">for</span> ref in dataRef), eName, e) AttributeError: <span class="code-quote">'list'</span> object has no attribute <span class="code-quote">'dataId'</span> </pre> </div></div> <p>There are a few problems here. Firstly, the root cause (the <tt>ValueError</tt>) is due to attempting to do 1d sky subtraction with no sky fibers. I've put in some code to catch that case and give a more informative error message.</p> <p>The extra exception (the <tt>AttributeError</tt>) is a common problem in the LSST Gen2 middleware, where the input to the <tt>runDataRef</tt> method doesn't match one of the types it thinks you might give it. Fixing it requires copying 50 lines of code to make a small fix, which doesn't seem worth it to me because it doesn't hide the true problem, and we're going to update to the Gen3 middleware.</p> <p>Finally, if we disable the 1d sky subtraction (<tt>-c doSubtractSky1d=False</tt>), we encounter a different problem: the fiberIds for the red and blue arms differ. This is likely because the detectorMaps are not accurate (I don't see any bootstrap products in <tt>/gpfs02/work/yasuda/PFS/calibs_Jul_<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-523" title="Demonstrate R, B and NIR arm merging" class="issue-link" data-issue-key="PIPE2D-523"><del>PIPE2D-523</del></a>/rerun</tt>), but this problem has been lurking for a while. I've added some new code to <tt>constructFiberTrace</tt> to catch this case, along with a brute force workaround that will be used if you set <tt>-c forceFiberIds=True</tt>.</p> <p>With those fixes, I've got <tt>mergeArms</tt> working on Yasuda-san's data.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> (lsst-scipipe-1172c30) price@fe:/gpfs02/work/price/pipe2d-523 $ constructFiberTrace.py /gpfs02/work/yasuda/PFS/calibs_Jul_PIPE2D-523 --calib /gpfs02/work/yasuda/PFS/calibs_Jul_PIPE2D-523/CALIB --output fiberTrace --id visit=21122..21126 --cores 8 -c forceFiberIds=True (lsst-scipipe-1172c30) price@fe:/gpfs02/work/price/pipe2d-523 $ cp -r /gpfs02/work/yasuda/PFS/calibs_Jul_PIPE2D-523/CALIB . (lsst-scipipe-1172c30) price@fe:/gpfs02/work/price/pipe2d-523 $ cp fiberTrace/FIBERTRACE/pfsFiberTrace-2019-07-25-021122-* CALIB/FIBERTRACE/ (lsst-scipipe-1172c30) price@fe:/gpfs02/work/price/pipe2d-523 $ reduceExposure.py /gpfs02/work/yasuda/PFS/calibs_Jul_PIPE2D-523 --calib CALIB --output=demo --id visit=21400 (lsst-scipipe-1172c30) price@fe:/gpfs02/work/price/pipe2d-523 $ mergeArms.py demo --calib CALIB --rerun merge --id visit=21400 -c doSubtractSky1d=False </pre> </div></div> <p>Merged to master.</p> Relates PIPE2D-1114 SIM2D-123 PIPE2D-316 Development Rank 0|02qprz:0i Reviewers hassan Sprint 2DDRP-2021 A Story Points 4.0