[PIPE2D-1259] fitContinuum: Make binning faster Created: 18/Jul/23  Updated: 02/Aug/23  Resolved: 02/Aug/23

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Normal
Reporter: sogo.mineo Assignee: sogo.mineo
Resolution: Done Votes: 0
Labels: flux-calibration
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Reviewers: price

 Description   

FitContinuumTask is called by FitPfsFluxReferenceTask very many times, to the extent that its execution time amounts to one of the largest part of FitPfsFluxReferenceTask's execution time. The major part of FitContinuumTask's execution time is from binData(). We processed visit=92114 to get the following profile:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.014    0.014 5329.600 5329.600 fitPfsFluxReference.py:292(run)
    38657    8.654    0.000 1604.840    0.042 fitContinuum.py:83(fitContinuum)
   154628   59.177    0.000 1002.344    0.006 fitContinuum.py:264(binData)

We can make binData() faster by calling np.median() only once. Here is the profile after this modification:

        1    0.019    0.019 4658.235 4658.235 fitPfsFluxReference.py:292(run)
    38657    8.498    0.000  942.938    0.024 fitContinuum.py:88(fitContinuum)
   154628   11.449    0.000  361.209    0.002 fitContinuum.py:271(binData)

10 minites/visit is saved by this modification with virtually no impact on outputs of the pipeline. I say "virtually" because I use a binning strategy different from the original code when numBins does not divide the array length. In many cases my binning agrees with the original binning, but they are different when, say, array length = 100 and numBins = 8. binData(np.arange(100).astype(float), numBins=8) produces the following results:

original: [ 6.  18.5 31.  43.5 56.  68.5 81.  93.5]
mine:     [ 6.  18.5 31.  43.5 55.5 68.  80.5 93. ]

I believe my one is better because the elements of "*.0" and "*.5" are distributed symmetrically in its output.



 Comments   
Comment by sogo.mineo [ 18/Jul/23 ]

Could you review this PR?

Comment by sogo.mineo [ 02/Aug/23 ]

Merged. Thanks for reviewing.

Comment by sogo.mineo [ 02/Aug/23 ]

I close this ticket. (I forgot to close this ticket but just commented a moment ago)

Generated at Sat Feb 10 16:05:10 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.