[PIPE2D-759] Speed up makeFiberTracesFromDetectorMap Created: 07/Mar/21 Updated: 18/Jun/21 Resolved: 18/Jun/21 |
|
| Status: | Done |
| Project: | DRP 2-D Pipeline |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Story | Priority: | Normal |
| Reporter: | rhl | Assignee: | price |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Story Points: | 2 |
| Sprint: | 2DDRP-2021 A3, 2DDRP-2021 A5, 2DDRP-2021 A 6 |
| Reviewers: | hassan |
| Description |
|
Running makeFiberTracesFromDetectorMap takes 90-100s, a very significant part of running reduceExposure.py. Please optimise the code, moving it to C++ if needs be. While processing a lot of data amortises this cost, for quick look on the mountain one is typically processing a visit at a time, and we cannot afford to wait so long. I have added checks in reduceExposureTask so we don't pay this cost when simply running ISR, but that is not sufficient for e.g. plotSuNSSFluxes.
|
| Comments |
| Comment by price [ 16/Jun/21 ] |
|
Time to beat is 49.390 sec.
(lsst-scipipe) pprice@tiger2-sumire:~/pfs/drp_stella[master] $ python -m cProfile -o pipe2d-759-before.profile $(which reduceExposure.py) /projects/HSC/PFS/Subaru --calib /scratch/pprice/pipe2d-829/CALIB --rerun price/pipe2d-759 --id visit=46782 arm=r -c isr.doFlat=False isr.doDefect=False isr.doDark=False repair.interp.modelPsf.defaultFwhm=2.25 --no-versions
(lsst-scipipe) pprice@tiger2-sumire:~/pfs/drp_stella[master%] $ python -c 'from pstats import Stats; Stats("pipe2d-759-before.profile").sort_stats("cumulative").print_stats(30)'
Tue Jun 15 16:00:57 2021 pipe2d-759-before.profile
161637507 function calls (157515421 primitive calls) in 226.881 seconds
Ordered by: cumulative time
List reduced from 9213 to 30 due to restriction <30>
ncalls tottime percall cumtime percall filename:lineno(function)
1808/1 0.026 0.000 226.882 226.882 {built-in method builtins.exec}
1 0.000 0.000 226.882 226.882 /home/pprice/pfs/drp_stella/bin/reduceExposure.py:23(<module>)
1 0.000 0.000 223.812 223.812 /tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py:548(parseAndRun)
1 0.099 0.099 222.879 222.879 /tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py:183(run)
1 0.000 0.000 222.774 222.774 /tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py:341(__call__)
1 0.048 0.048 222.773 222.773 /tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py:428(runTask)
1 0.000 0.000 222.725 222.725 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/reduceExposure.py:166(runDataRef)
1 0.051 0.051 216.054 216.054 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/reduceExposure.py:402(getSpectralCalibs)
1 0.005 0.005 94.666 94.666 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/photometerLines.py:56(run)
1349/1085 0.019 0.000 92.073 0.085 {built-in method builtins.next}
4 0.049 0.012 92.030 23.008 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:260(subtractionContext)
72 0.000 0.000 91.995 1.278 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/contextlib.py:107(__enter__)
2 1.547 0.774 91.981 45.990 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:228(subtractContinuum)
2 0.007 0.004 89.858 44.929 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:54(run)
502 0.104 0.000 89.827 0.179 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:81(fitContinuum)
502 57.959 0.115 82.754 0.165 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:195(maskLines)
2 0.000 0.000 49.390 24.695 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fiberProfileSet.py:119(makeFiberTracesFromDetectorMap)
2 0.018 0.009 49.208 24.604 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fiberProfileSet.py:137(makeFiberTraces)
502 19.232 0.038 49.190 0.098 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fiberProfile.py:228(makeFiberTrace)
1 0.022 0.022 40.013 40.013 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/adjustDetectorMap.py:65(run)
4 0.019 0.005 39.337 9.834 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/adjustDetectorMap.py:254(fit)
4 0.000 0.000 39.006 9.752 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/_minimize.py:42(minimize)
4 0.001 0.000 39.006 9.751 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:2509(_minimize_powell)
518 0.443 0.001 38.988 0.075 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:298(function_wrapper)
518 1.392 0.003 38.545 0.074 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/adjustDetectorMap.py:219(calculateChi2)
55 0.001 0.000 38.326 0.697 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:2364(_linesearch_powell)
55 0.000 0.000 38.325 0.697 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:2011(brent)
55 0.001 0.000 38.324 0.697 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:2090(_minimize_scalar_brent)
55 0.010 0.000 38.323 0.697 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:1896(optimize)
509 0.004 0.000 38.313 0.075 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:2370(myfunc)
|
| Comment by price [ 17/Jun/21 ] |
|
Got it working in C++, but with worse precision: test_profileNorm.py is failing because the residuals of the extracted spectra have a standard deviation of about 3.5e-8 instead of 1.5e-8. But the time has been reduced to 1.547 sec, almost a 32x reduction! I'm going to see if I can find an obvious cause of the loss of precision, but it's not too awful so I'm not sure it's worth a lot of extra effort. |
| Comment by price [ 17/Jun/21 ] |
|
Found the problem, and we're passing all tests now. Following this change, tall poles are now the continuum subtraction and detectorMap adjustment:
(lsst-scipipe) pprice@tiger2-sumire:~/pfs/drp_stella[tickets/PIPE2D-759%] $ python -c 'from pstats import Stats; Stats("pipe2d-759-after.profile").sort_stats("cumulative").print_stats(30)'
Wed Jun 16 15:23:19 2021 pipe2d-759-after.profile
156838987 function calls (152716719 primitive calls) in 177.606 seconds
Ordered by: cumulative time
List reduced from 9208 to 30 due to restriction <30>
ncalls tottime percall cumtime percall filename:lineno(function)
1822/1 0.026 0.000 177.607 177.607 {built-in method builtins.exec}
1 0.000 0.000 177.607 177.607 /home/pprice/pfs/drp_stella/bin/reduceExposure.py:23(<module>)
1 0.000 0.000 174.097 174.097 /tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py:548(parseAndRun)
1 0.076 0.076 172.896 172.896 /tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py:183(run)
1 0.000 0.000 172.698 172.698 /tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py:341(__call__)
1 0.049 0.049 172.697 172.697 /tigress/HSC/PFS/stack/20190925/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/18.1.0/python/lsst/pipe/base/cmdLineTask.py:428(runTask)
1 0.000 0.000 172.649 172.649 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/reduceExposure.py:166(runDataRef)
1 0.063 0.063 166.064 166.064 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/reduceExposure.py:402(getSpectralCalibs)
1 0.004 0.004 92.969 92.969 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/photometerLines.py:56(run)
1349/1086 0.021 0.000 90.462 0.083 {built-in method builtins.next}
70 0.000 0.000 90.353 1.291 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/contextlib.py:107(__enter__)
4 0.049 0.012 90.329 22.582 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:260(subtractionContext)
2 1.461 0.730 90.280 45.140 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:228(subtractContinuum)
2 0.007 0.004 88.150 44.075 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:54(run)
502 0.109 0.000 88.119 0.176 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:81(fitContinuum)
502 56.832 0.113 80.974 0.161 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:195(maskLines)
1 0.025 0.025 39.220 39.220 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/adjustDetectorMap.py:65(run)
4 0.020 0.005 38.515 9.629 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/adjustDetectorMap.py:254(fit)
4 0.000 0.000 38.190 9.548 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/_minimize.py:42(minimize)
4 0.001 0.000 38.190 9.547 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:2509(_minimize_powell)
518 0.400 0.001 38.173 0.074 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:298(function_wrapper)
518 1.366 0.003 37.773 0.073 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/adjustDetectorMap.py:219(calculateChi2)
55 0.001 0.000 37.521 0.682 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:2364(_linesearch_powell)
55 0.000 0.000 37.520 0.682 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:2011(brent)
55 0.001 0.000 37.519 0.682 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:2090(_minimize_scalar_brent)
55 0.009 0.000 37.518 0.682 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:1896(optimize)
509 0.004 0.000 37.508 0.074 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/site-packages/scipy/optimize/optimize.py:2370(myfunc)
1 0.097 0.097 29.446 29.446 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/centroidLines.py:72(run)
1 0.000 0.000 24.276 24.276 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/centroidLines.py:109(centroidLines)
522 8.448 0.016 22.877 0.044 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/adjustDetectorMap.py:201(calculateLineResiduals)
|
| Comment by hassan [ 18/Jun/21 ] |
|
Updates look fine. |
| Comment by price [ 18/Jun/21 ] |
|
Merged. |