https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-839 DRP 2-D Pipeline <p><tt>AdjustDetectorMapTask</tt> uses a low-order non-linear fit to adjust the detectorMap using the traces and lines, but this is slower than we would like (~ 70 sec for SuNSS on tiger). If we're close to the solution (as we should be if we're just doing an adjustment), then we should be able to switch to a linear fit with the traces converted to lines with a large y error, which would be much faster.</p> PIPE2D-839

Make AdjustDetectorMapTask faster

Story Normal Done Done price price Thu, 13 May 2021 19:42:11 +0000 Wed, 4 Aug 2021 01:57:58 +0000 Wed, 4 Aug 2021 01:57:58 +0000 0 2 <p>Also took the opportunity to deal with another tall pole (line masking in the continuum subtraction). These two together have reduced the runtime of <tt>reduceExposure.py</tt> from 213.352 sec to 82.923 sec (2.5x).</p> <p>Tall poles are now centroiding, photometry, the new improved adjustDetectorMap and continuum subtraction:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ncalls tottime percall cumtime percall filename:lineno(function) 1 0.087 0.087 29.473 29.473 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/centroidLines.py:72(run) 1 0.000 0.000 24.689 24.689 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/centroidLines.py:109(centroidLines) 1 0.005 0.005 17.456 17.456 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/photometerLines.py:56(run) 1 0.015 0.015 15.279 15.279 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/adjustDetectorMap.py:29(run) 2035/1764 0.012 0.000 13.718 0.008 {built-in method builtins.next} 71 0.000 0.000 13.630 0.192 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/contextlib.py:107(__enter__) 4 0.048 0.012 13.611 3.403 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:229(subtractionContext) 2 1.576 0.788 13.563 6.782 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:197(subtractContinuum) 1 0.026 0.026 11.837 11.837 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py:453(fitDetectorDistortion) </pre> </div></div> <p>By running on the weekly (600 fibers), SuNSS data (251 fibers; flat-fielding disabled) and the integration test (15 fibers), we get a sense of how the runtime scales:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> nFibers runTime 600 170 251 67 15 29 </pre> </div></div> <p>Changes look fine. It looks to me that one change is what is documented in the commits as a 'linear solver' where I see low-order terms being utilized. If I'm correct, the commits might need to be re-phrased a little.</p> <p>Merged.</p> Development Rank 0|zzs8a1: Reviewers hassan Sprint 2DDRP-2021 A 7