Moving the fluxmodeldata to a faster disk has alleviated the problem on tiger.

I took the opportunity to look at the fitPfsFluxReference code and made some observations that might help improve it moving forward:

• Reading the 9 GB fluxModelInterpolator pickle is done in the FitPfsFluxReferenceTask init, which means it's read once at the start, and lives until everything is done. Given that it's so large, it should probably be read, used and then dumped in order to free memory.
  • The interpolator pickle is large because it contains all the spectra. Presumably that means it contains all the information necessary to reproduce the model spectra, so is it necessary to read all the spectra as well?
  • All the spectra are going into the interpolation, even though many are strongly down-weighted. I suggest using the neighbours parameter of the RBFInterpolator.
• The calculations in FitPfsFluxReferenceTask.correctExtinction should be pre-computed: E(B-V) translates directly to A_X for each band X given R_X: A_X = R_X * E(B-V). So we can pre-compute the R_X and put it in the broadband photometry reference file.
• The fitting algorithm seems to be brute-force: it's mapping out the entire chi^2 space, which means it reads every spectrum, even though many are effectively eliminated by the photometry (e.g., some have a broad-band photometry probability smaller than 10^-200).

Stepping back from the details of the current implementation, I think that this would go much faster with a modified algorithm:
1. Fit Teff from broad-band photometry.
2. Measure logg,m,alpha from line indices. I confess I don't know exactly how this would work, but I think logg must be a function of the width of Balmer lines (and Paschen lines, if available), and m and alpha must be a function of the equivalent width of different metal lines.
3. Measure radial velocity using the best guess parameters Teff,logg,m,alpha.
4. Create an interpolator using the (RV-corrected, LSF-convolved, whitened) spectra within a hyper-sphere of the best guess parameters Teff,logg,m,alpha +/- twice their errors. I'm guessing that linear interpolation between the closest models is sufficient, but an RBF could be used if necessary.
5. Use a non-linear minimizer (e.g., Powell's method) with the interpolator to fit the spectrum. I think this should converge fairly quickly; if not, a grid search could be done as currently, but over a much-reduced subset of parameter space.

Thanks for the suggestions. I will think about them a few days later, once my urgent task is done.

> All the spectra are going into the interpolation, even though many are strongly down-weighted. I suggest using the neighbours parameter of the RBFInterpolator.

I have not tried it, but I don't think any parameter will help reduce memory usage. RBFInterpolator needs all the input spectra to be on memory by nature. Yamashita-san said that we might be able to construct RBFInterpolator every time it is required, from only a few spectra around the initial guess (Teff0, logg0, m0, alpha0) – if we can find the initial guess accurately.

Thank you for your suggestions about the algorithm! Mineo-san has implemented the cut-off threshold to narrow down the models after broad-band fitting in PIPE2D-1145. We can use it for the neighbours parameter. We should try it, probably after the data release this month.

It is worth a try your idea using line indices and a minimizer. Stabilizing and optimizing the current flux calibration is a priority, but we may want to start a study of your algorithm.