[PIPE2D-1332] fitPfsFluxReference: Use 1D prior rather than 4D prior

Wed, 22 Nov 2023 01:34:31 +0000

fitPfsFluxReference: Use 1D prior rather than 4D prior

One of bottlenecks of fitPfsFluxReference has turned out to be
scipy.interpolate.LinearNDInterpolator called many times by the objective function.
There, scipy.interpolate.LinearNDInterpolator interpolates 4D prior probability density function.

Because the prior PDF is made from broadband fluxes, the prior PDF is not very meaningful except
along the axis of effective temperature. If we marginalize the other three dimensions, we can do
without this slow interpolator.

In fact, we can improve speed by using a 1D prior. Here is a profile before the modification:

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1    0.001    0.001 1958.247 1958.247 fitPfsFluxReference.py:302(runDataRef)
     1  105.904  105.904 1493.559 1493.559 fitPfsFluxReference.py:643(fitModelsToSpectra)
 15284  563.177    0.037 1386.914    0.091 fitPfsFluxReference.py:766(objective)

Here is a profile after the modification:

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1    0.004    0.004 1659.727 1659.727 fitPfsFluxReference.py:302(runDataRef)
     1    0.016    0.016 1197.156 1197.156 fitPfsFluxReference.py:643(fitModelsToSpectra)
 18526    3.910    0.000 1191.519    0.064 fitPfsFluxReference.py:771(objective)

Though ncalls of objective gets larger after the modification, its percall gets 30% less,
which leads to fitModelsToSpectra running faster by 300 seconds / visit.

Quality of fitting does not change. We input 100 known spectra to get the following results:

                               4D   ( % stddev)    1D   ( % stddev)
----------------------------------------------------------------------
   RMS of errors of  'teff'  216.03 (   29.35 )  216.76 (   29.45 )
   RMS of errors of  'logg'    0.58 (   52.03 )    0.44 (   39.20 )
   RMS of errors of     'm'    0.99 (   62.27 )    0.86 (   53.76 )
   RMS of errors of 'alpha'    0.33 (   98.34 )    0.36 (  109.98 )
   RMS of errors of   'vel'   49.31 (  inf    )   49.31 (  inf    )
----------------------------------------------------------------------
  Mean of errors of  'teff' -148.32 (  -20.15 ) -144.36 (  -19.61 )
  Mean of errors of  'logg'    0.13 (   11.58 )    0.09 (    8.07 )
  Mean of errors of     'm'   -0.40 (  -24.86 )   -0.43 (  -26.87 )
  Mean of errors of 'alpha'    0.08 (   25.11 )    0.05 (   15.41 )
  Mean of errors of   'vel'   -1.75 ( -inf    )   -1.75 ( -inf    )

(For example, RMS of errors of 'teff' (in K) is sqrt of \frac{1}{n}\sum_{i} (teff^{out}_{i} - teff^{in}_{i})^2.
"% stddev" means this value over the standard deviation of {{teff^{in}_{i} }}.)

PFS-JIRA

[PIPE2D-1332] fitPfsFluxReference: Use 1D prior rather than 4D prior