https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-759 DRP 2-D Pipeline <p>Running <tt>makeFiberTracesFromDetectorMap</tt> takes 90-100s, a very significant part of running <tt>reduceExposure.py</tt>.  Please optimise the code, moving it to C++ if needs be.</p> <p>While processing a lot of data amortises this cost, for quick look on the mountain one is typically processing a visit at a time, and we cannot afford to wait so long.  I have added checks in <tt>reduceExposureTask</tt> so we don't pay this cost when simply running ISR, but that is not sufficient for e.g. <tt>plotSuNSSFluxes</tt>.</p> <p> </p> PIPE2D-759

Speed up makeFiberTracesFromDetectorMap

Story Normal Done Done price rhl Sat, 6 Mar 2021 20:25:33 +0000 Thu, 17 Jun 2021 19:38:26 +0000 Thu, 17 Jun 2021 19:38:26 +0000 0 3 <p>Time to beat is 49.390 sec.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> (lsst-scipipe) pprice@tiger2-sumire:~/pfs/drp_stella[master] $ python -m cProfile -o pipe2d-759-before.profile $(which reduceExposure.py) /projects/HSC/PFS/Subaru --calib /scratch/pprice/pipe2d-829/CALIB --rerun price/pipe2d-759 --id visit=46782 arm=r -c isr.doFlat=False isr.doDefect=False isr.doDark=False repair.interp.modelPsf.defaultFwhm=2.25 --no-versions (lsst-scipipe) pprice@tiger2-sumire:~/pfs/drp_stella[master%] $ python -c <span class="code-quote">'from pstats <span class="code-keyword">import</span> Stats; Stats(<span class="code-quote">"pipe2d-759-before.profile"</span>).sort_stats(<span class="code-quote">"cumulative"</span>).print_stats(30)'</span> Tue Jun 15 16:00:57 2021 pipe2d-759-before.profile 161637507 function calls (157515421 primitive calls) in 226.881 seconds Ordered by: cumulative time List reduced from 9213 to 30 due to restriction &lt;30&gt; ncalls tottime percall cumtime percall filename:lineno(function) 1808/1 0.026 0.000 226.882 226.882 {built-in method builtins.exec} 1 0.000 0.000 226.882 226.882 /home/pprice/pfs/drp_stella/bin/reduceExposure.py:23(&lt;module&gt;) 1 0.000 0.000 223.812 223.812 /tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py:548(parseAndRun) 1 0.099 0.099 222.879 222.879 /tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py:183(run) 1 0.000 0.000 222.774 222.774 /tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py:341(__call__) 1 0.048 0.048 222.773 222.773 /tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py:428(runTask) 1 0.000 0.000 222.725 222.725 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/reduceExposure.py:166(runDataRef) 1 0.051 0.051 216.054 216.054 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/reduceExposure.py:402(getSpectralCalibs) 1 0.005 0.005 94.666 94.666 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/photometerLines.py:56(run) 1349/1085 0.019 0.000 92.073 0.085 {built-in method builtins.next} 4 0.049 0.012 92.030 23.008 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:260(subtractionContext) 72 0.000 0.000 91.995 1.278 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/contextlib.py:107(__enter__) 2 1.547 0.774 91.981 45.990 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:228(subtractContinuum) 2 0.007 0.004 89.858 44.929 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:54(run) 502 0.104 0.000 89.827 0.179 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:81(fitContinuum) 502 57.959 0.115 82.754 0.165 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:195(maskLines) 2 0.000 0.000 49.390 24.695 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fiberProfileSet.py:119(makeFiberTracesFromDetectorMap) 2 0.018 0.009 49.208 24.604 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fiberProfileSet.py:137(makeFiberTraces) 502 19.232 0.038 49.190 0.098 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fiberProfile.py:228(makeFiberTrace) 1 0.022 0.022 40.013 40.013 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/adjustDetectorMap.py:65(run) 4 0.019 0.005 39.337 9.834 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/adjustDetectorMap.py:254(fit) 4 0.000 0.000 39.006 9.752 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/_minimize.py:42(minimize) 4 0.001 0.000 39.006 9.751 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:2509(_minimize_powell) 518 0.443 0.001 38.988 0.075 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:298(function_wrapper) 518 1.392 0.003 38.545 0.074 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/adjustDetectorMap.py:219(calculateChi2) 55 0.001 0.000 38.326 0.697 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:2364(_linesearch_powell) 55 0.000 0.000 38.325 0.697 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:2011(brent) 55 0.001 0.000 38.324 0.697 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:2090(_minimize_scalar_brent) 55 0.010 0.000 38.323 0.697 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:1896(optimize) 509 0.004 0.000 38.313 0.075 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:2370(myfunc) </pre> </div></div> <p>Got it working in C++, but with worse precision: <tt>test_profileNorm.py</tt> is failing because the residuals of the extracted spectra have a standard deviation of about 3.5e-8 instead of 1.5e-8. But the time has been reduced to 1.547 sec, almost a 32x reduction!</p> <p>I'm going to see if I can find an obvious cause of the loss of precision, but it's not too awful so I'm not sure it's worth a lot of extra effort.</p> <p>Found the problem, and we're passing all tests now.</p> <p>Following this change, tall poles are now the continuum subtraction and detectorMap adjustment:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> (lsst-scipipe) pprice@tiger2-sumire:~/pfs/drp_stella[tickets/PIPE2D-759%] $ python -c <span class="code-quote">'from pstats <span class="code-keyword">import</span> Stats; Stats(<span class="code-quote">"pipe2d-759-after.profile"</span>).sort_stats(<span class="code-quote">"cumulative"</span>).print_stats(30)'</span> Wed Jun 16 15:23:19 2021 pipe2d-759-after.profile 156838987 function calls (152716719 primitive calls) in 177.606 seconds Ordered by: cumulative time List reduced from 9208 to 30 due to restriction &lt;30&gt; ncalls tottime percall cumtime percall filename:lineno(function) 1822/1 0.026 0.000 177.607 177.607 {built-in method builtins.exec} 1 0.000 0.000 177.607 177.607 /home/pprice/pfs/drp_stella/bin/reduceExposure.py:23(&lt;module&gt;) 1 0.000 0.000 174.097 174.097 /tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py:548(parseAndRun) 1 0.076 0.076 172.896 172.896 /tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py:183(run) 1 0.000 0.000 172.698 172.698 /tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py:341(__call__) 1 0.049 0.049 172.697 172.697 /tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py:428(runTask) 1 0.000 0.000 172.649 172.649 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/reduceExposure.py:166(runDataRef) 1 0.063 0.063 166.064 166.064 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/reduceExposure.py:402(getSpectralCalibs) 1 0.004 0.004 92.969 92.969 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/photometerLines.py:56(run) 1349/1086 0.021 0.000 90.462 0.083 {built-in method builtins.next} 70 0.000 0.000 90.353 1.291 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/contextlib.py:107(__enter__) 4 0.049 0.012 90.329 22.582 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:260(subtractionContext) 2 1.461 0.730 90.280 45.140 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:228(subtractContinuum) 2 0.007 0.004 88.150 44.075 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:54(run) 502 0.109 0.000 88.119 0.176 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:81(fitContinuum) 502 56.832 0.113 80.974 0.161 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:195(maskLines) 1 0.025 0.025 39.220 39.220 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/adjustDetectorMap.py:65(run) 4 0.020 0.005 38.515 9.629 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/adjustDetectorMap.py:254(fit) 4 0.000 0.000 38.190 9.548 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/_minimize.py:42(minimize) 4 0.001 0.000 38.190 9.547 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:2509(_minimize_powell) 518 0.400 0.001 38.173 0.074 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:298(function_wrapper) 518 1.366 0.003 37.773 0.073 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/adjustDetectorMap.py:219(calculateChi2) 55 0.001 0.000 37.521 0.682 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:2364(_linesearch_powell) 55 0.000 0.000 37.520 0.682 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:2011(brent) 55 0.001 0.000 37.519 0.682 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:2090(_minimize_scalar_brent) 55 0.009 0.000 37.518 0.682 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:1896(optimize) 509 0.004 0.000 37.508 0.074 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:2370(myfunc) 1 0.097 0.097 29.446 29.446 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/centroidLines.py:72(run) 1 0.000 0.000 24.276 24.276 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/centroidLines.py:109(centroidLines) 522 8.448 0.016 22.877 0.044 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/adjustDetectorMap.py:201(calculateLineResiduals) </pre> </div></div> <p>Updates look fine.</p> <p>Merged.</p> Development Rank 0|02qpt6:508yl0chzzy00000zj Reviewers hassan Sprint 2DDRP-2021 A3 2DDRP-2021 A5 2DDRP-2021 A 6 Story Points 2.0