https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1168 DRP 2-D Pipeline <p>Running <tt>fitPfsReferenceFlux</tt> on our Tiger head node trashes the machine. I am reducing the new weekly:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> fitPfsReferenceFlux.py /scratch/pprice/sim2d-151/test --calib=/scratch/pprice/sim2d-151/test/CALIB --rerun=weekly/pipeline/brn/pipeline --doraise -j10 --no-versions --clobber-config --id <span class="code-quote">'visit=47^48^49^53^54^57'</span> <span class="code-quote">'arm=b^r^n'</span> </pre> </div></div> <p>I suspect that the <tt>fluxmodeldata</tt> is being accessed within a tight loop, which causes a huge load on the disk.</p> PIPE2D-1168

fitPfsReferenceFlux breaks Tiger

Story Normal Done Done price price Tue, 21 Feb 2023 19:40:20 +0000 Fri, 16 Jun 2023 06:56:30 +0000 Tue, 28 Feb 2023 19:33:32 +0000 0 4 <p>Moving the <tt>fluxmodeldata</tt> to a faster disk has alleviated the problem on tiger.</p> <p>I took the opportunity to look at the <tt>fitPfsFluxReference</tt> code and made some observations that might help improve it moving forward:</p> <ul> <li>Reading the 9 GB <tt>fluxModelInterpolator</tt> pickle is done in the <tt>FitPfsFluxReferenceTask</tt> init, which means it's read once at the start, and lives until everything is done. Given that it's so large, it should probably be read, used and then dumped in order to free memory. <ul> <li>The interpolator pickle is large because it contains all the spectra. Presumably that means it contains all the information necessary to reproduce the model spectra, so is it necessary to read all the spectra as well?</li> <li>All the spectra are going into the interpolation, even though many are strongly down-weighted. I suggest using the <tt>neighbours</tt> parameter of the <tt>RBFInterpolator</tt>.</li> </ul> </li> <li>The calculations in <tt>FitPfsFluxReferenceTask.correctExtinction</tt> should be pre-computed: E(B-V) translates directly to A_X for each band X given R_X: A_X = R_X * E(B-V). So we can pre-compute the R_X and put it in the broadband photometry reference file.</li> <li>The fitting algorithm seems to be brute-force: it's mapping out the entire chi^2 space, which means it reads every spectrum, even though many are effectively eliminated by the photometry (e.g., some have a broad-band photometry probability smaller than 10^-200).</li> </ul> <p>Stepping back from the details of the current implementation, I think that this would go much faster with a modified algorithm:<br/> 1. Fit Teff from broad-band photometry.<br/> 2. Measure logg,m,alpha from line indices. I confess I don't know exactly how this would work, but I think logg must be a function of the width of Balmer lines (and Paschen lines, if available), and m and alpha must be a function of the equivalent width of different metal lines.<br/> 3. Measure radial velocity using the best guess parameters Teff,logg,m,alpha.<br/> 4. Create an interpolator using the (RV-corrected, LSF-convolved, whitened) spectra within a hyper-sphere of the best guess parameters Teff,logg,m,alpha +/- twice their errors. I'm guessing that linear interpolation between the closest models is sufficient, but an RBF could be used if necessary.<br/> 5. Use a non-linear minimizer (e.g., Powell's method) with the interpolator to fit the spectrum. I think this should converge fairly quickly; if not, a grid search could be done as currently, but over a much-reduced subset of parameter space.</p> <p>Thanks for the suggestions. I will think about them a few days later, once my urgent task is done.</p> <p>&gt; All the spectra are going into the interpolation, even though many are strongly down-weighted. I suggest using the neighbours parameter of the RBFInterpolator.</p> <p>I have not tried it, but I don't think any parameter will help reduce memory usage. RBFInterpolator needs all the input spectra to be on memory by nature. Yamashita-san said that we might be able to construct RBFInterpolator every time it is required, from only a few spectra around the initial guess (Teff0, logg0, m0, alpha0) – if we can find the initial guess accurately.</p> <p>Thank you for your suggestions about the algorithm! Mineo-san has implemented the cut-off threshold to narrow down the models after broad-band fitting in <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1145" title="Make fitPfsFluxReference.py faster" class="issue-link" data-issue-key="PIPE2D-1145"><del>PIPE2D-1145</del></a>. We can use it for the neighbours parameter. We should try it, probably after the data release this month.</p> <p>It is worth a try your idea using line indices and a minimizer. Stabilizing and optimizing the current flux calibration is a priority, but we may want to start a study of your algorithm.</p> <p> </p> Blocks SIM2D-151 Relates PIPE2D-1208 PIPE2D-1231 PIPE2D-1238 Development Rank 0|zzsw9z: