[PIPE2D-839] Make AdjustDetectorMapTask faster Created: 14/May/21 Updated: 04/Aug/21 Resolved: 04/Aug/21 |
|
| Status: | Done |
| Project: | DRP 2-D Pipeline |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Story | Priority: | Normal |
| Reporter: | price | Assignee: | price |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Sprint: | 2DDRP-2021 A 7 |
| Reviewers: | hassan |
| Description |
|
AdjustDetectorMapTask uses a low-order non-linear fit to adjust the detectorMap using the traces and lines, but this is slower than we would like (~ 70 sec for SuNSS on tiger). If we're close to the solution (as we should be if we're just doing an adjustment), then we should be able to switch to a linear fit with the traces converted to lines with a large y error, which would be much faster. |
| Comments |
| Comment by price [ 28/Jul/21 ] |
|
Also took the opportunity to deal with another tall pole (line masking in the continuum subtraction). These two together have reduced the runtime of reduceExposure.py from 213.352 sec to 82.923 sec (2.5x). Tall poles are now centroiding, photometry, the new improved adjustDetectorMap and continuum subtraction:
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.087 0.087 29.473 29.473 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/centroidLines.py:72(run)
1 0.000 0.000 24.689 24.689 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/centroidLines.py:109(centroidLines)
1 0.005 0.005 17.456 17.456 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/photometerLines.py:56(run)
1 0.015 0.015 15.279 15.279 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/adjustDetectorMap.py:29(run)
2035/1764 0.012 0.000 13.718 0.008 {built-in method builtins.next}
71 0.000 0.000 13.630 0.192 /tigress/HSC/PFS/stack/20190925/python/miniconda3-4.5.12/envs/lsst-scipipe/lib/python3.7/contextlib.py:107(__enter__)
4 0.048 0.012 13.611 3.403 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:229(subtractionContext)
2 1.576 0.788 13.563 6.782 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitContinuum.py:197(subtractContinuum)
1 0.026 0.026 11.837 11.837 /home/pprice/pfs/drp_stella/python/pfs/drp/stella/fitDistortedDetectorMap.py:453(fitDetectorDistortion)
By running on the weekly (600 fibers), SuNSS data (251 fibers; flat-fielding disabled) and the integration test (15 fibers), we get a sense of how the runtime scales: nFibers runTime 600 170 251 67 15 29 |
| Comment by hassan [ 04/Aug/21 ] |
|
Changes look fine. It looks to me that one change is what is documented in the commits as a 'linear solver' where I see low-order terms being utilized. If I'm correct, the commits might need to be re-phrased a little. |
| Comment by price [ 04/Aug/21 ] |
|
Merged. |