[PIPE2D-1209] Optimize reduceProfiles Created: 29/Apr/23 Updated: 17/May/23 Resolved: 17/May/23 |
|
| Status: | Done |
| Project: | DRP 2-D Pipeline |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Story | Priority: | Normal |
| Reporter: | price | Assignee: | price |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Description |
|
reduceProfiles.py is peaking at 70% of memory use per spectrograph arm on machines at Hilo when running on the latest data. It's also taking a few hours per spectrograph arm. I'd like to improve both of these aspects. reduceProfiles.py /work/drp --calib=/work/drp/CALIB --rerun=price/20230428-profiles --doraise --id visit=91808^91837^91838^91840^91841^91843^91844^91907..91909^91936..91940^91942..91946^91948..91952 arm=r spectrograph=1 --normId visit=91907..91909 arm=r spectrograph=1 --no-versions -c reduceExposure.doAdjustDetectorMap=True profiles.profileRadius=10 profiles.profileOversample=3 profiles.profileSwath=2000 profiles.profileRejThresh=5 reduceExposure.isr.doFlat=False --no-versions -j 20 |
| Comments |
| Comment by price [ 17/May/23 ] |
|
Memory use was drastically reduced (70% --> about 3%). Didn't make significant progress on speed, but I'm not too concerned about that right now since creating fiber profiles isn't a frequent operation. Merged to master. |