PFS-JIRA https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 [PIPE2D-365] Interpolate between model templates https://pfspipe.ipmu.jp/jira/browse/PIPE2D-365 DRP 2-D Pipeline <p>Provide a means of interpolating the AMBRE-derived model template data such that accurate model spectra for the candidate F-stars are determined.</p> <p>The source code for the interpolation, along with documentation, should be provided under GitHub. </p> PIPE2D-365 Interpolate between model templates Story Normal Done Done Takuji Yamashita hassan Tue, 19 Feb 2019 16:51:19 +0000 Mon, 20 Dec 2021 02:59:55 +0000 Mon, 20 Dec 2021 02:59:55 +0000 0 5 <p>I tried interpolations using some methods (1D, 3D Linear, 1D RBF, 3D RBF, 4D RBF) and compared the results with a reference model. The interpolations were done at a parameter point where we have a model.</p> <p> </p> <h4><a name="1D"></a>1D</h4> <p>Results of 1D interpolation in the direction of the effective temperature (Teff).  The cubic spline interpolation is the best among the test cases. The accuracies are low around 400nm for all the cases.<br/> <span class="image-wrap" style=""><a id="12036_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12036/12036_Linear3D_interpolation_residual_380-420.png" title="Linear3D_interpolation_residual_380-420.png" file-preview-type="image" file-preview-id="12036" file-preview-title="Linear3D_interpolation_residual_380-420.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12032/_thumb_12032.png" style="border: 0px solid black" /></a></span><span class="image-wrap" style=""><a id="12037_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12037/12037_1d_rbf_results_residual_7cases_example_epsilon2000.png" title="1d_rbf_results_residual_7cases_example_epsilon2000.png" file-preview-type="image" file-preview-id="12037" file-preview-title="1d_rbf_results_residual_7cases_example_epsilon2000.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12033/_thumb_12033.png" style="border: 0px solid black" /></a></span><span class="image-wrap" style=""><a id="12037_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12037/12037_1d_rbf_results_residual_7cases_example_epsilon2000.png" title="1d_rbf_results_residual_7cases_example_epsilon2000.png" file-preview-type="image" file-preview-id="12037" file-preview-title="1d_rbf_results_residual_7cases_example_epsilon2000.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12034/_thumb_12034.png" style="border: 0px solid black" /></a></span><br/>  </p> <h4><a name="3DLinear"></a>3D Linear</h4> <p>A result of the interpolation with 3 parameters (Teff, log(g), and metallicity). The residual is 0.4 % (standard dev) for the full wavelength. The accuracy around 400nm still remains low (1.4% at 390-400nm).<br/> <span class="image-wrap" style=""><a id="12039_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12039/12039_RBF3D_interpolation_MQepsilon2000_residual.png" title="RBF3D_interpolation_MQepsilon2000_residual.png" file-preview-type="image" file-preview-id="12039" file-preview-title="RBF3D_interpolation_MQepsilon2000_residual.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12035/_thumb_12035.png" style="border: 0px solid black" /></a></span><span class="image-wrap" style=""><a id="12040_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12040/12040_RBF3D_interpolation_MQepsilon2000_residual_yzoom.png" title="RBF3D_interpolation_MQepsilon2000_residual_yzoom.png" file-preview-type="image" file-preview-id="12040" file-preview-title="RBF3D_interpolation_MQepsilon2000_residual_yzoom.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12036/_thumb_12036.png" style="border: 0px solid black" /></a></span><br/>  <br/> I have not yet done the spline interpolations with &gt;3 parameters because no module for higher dimensions is available. Now I am trying to do that by separating interpolations into each one parameter.<br/>  </p> <h4><a name="1DRBF"></a>1D RBF</h4> <p>The radial basis function interpolation (scipy.interpolate.Rbf). Multiquadric, Inverse multiquadric, Gaussian, Thin plate reproduce the reference model well.<br/> <span class="image-wrap" style=""><a id="12041_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12041/12041_RBF3D_interpolation_MQepsilon2000_residual_380-420.png" title="RBF3D_interpolation_MQepsilon2000_residual_380-420.png" file-preview-type="image" file-preview-id="12041" file-preview-title="RBF3D_interpolation_MQepsilon2000_residual_380-420.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12037/_thumb_12037.png" style="border: 0px solid black" /></a></span><span class="image-wrap" style=""><a id="12042_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12042/12042_RBF4D_interpolation_MQepsilon1000_residual.png" title="RBF4D_interpolation_MQepsilon1000_residual.png" file-preview-type="image" file-preview-id="12042" file-preview-title="RBF4D_interpolation_MQepsilon1000_residual.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12038/_thumb_12038.png" style="border: 0px solid black" /></a></span></p> <h4><a name="3DRBF"></a>3D RBF</h4> <p>RBF interpolation with 3 parameters (Teff, log(g) and metallicity) using the multiquadric kernels as a test. The standard deviation of the residual is 0.2 % for the full wavelength and 1.2% at 390-400nm. This is better than the 3D linear interpolation. <br/> <span class="image-wrap" style=""><a id="12041_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12041/12041_RBF3D_interpolation_MQepsilon2000_residual_380-420.png" title="RBF3D_interpolation_MQepsilon2000_residual_380-420.png" file-preview-type="image" file-preview-id="12041" file-preview-title="RBF3D_interpolation_MQepsilon2000_residual_380-420.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12039/_thumb_12039.png" style="border: 0px solid black" /></a></span><span class="image-wrap" style=""><a id="12042_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12042/12042_RBF4D_interpolation_MQepsilon1000_residual.png" title="RBF4D_interpolation_MQepsilon1000_residual.png" file-preview-type="image" file-preview-id="12042" file-preview-title="RBF4D_interpolation_MQepsilon1000_residual.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12040/_thumb_12040.png" style="border: 0px solid black" /></a></span><span class="image-wrap" style=""><a id="12107_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12107/12107_SBS_Teff_logg_400nm.png" title="SBS_Teff_logg_400nm.png" file-preview-type="image" file-preview-id="12107" file-preview-title="SBS_Teff_logg_400nm.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12041/_thumb_12041.png" style="border: 0px solid black" /></a></span><br/>  </p> <h4><a name="4DRBF"></a>4D RBF</h4> <p>It is under investigation but a current RBF interpolation with 4 parameters including <span class="error">&#91;alpha/Fe&#93;</span> is not good.<br/> <span class="image-wrap" style=""><a id="12109_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12109/12109_RBF3D_Teffd1000_original.png" title="RBF3D_Teffd1000_original.png" file-preview-type="image" file-preview-id="12109" file-preview-title="RBF3D_Teffd1000_original.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12042/_thumb_12042.png" style="border: 0px solid black" /></a></span><br/>  <br/>  <br/>  <br/>  </p> <h3><a name="Stepbystepinterpolation"></a><b>Step-by-step interpolation</b></h3> <p>I tried a 2D interpolation in a step-by-step manner (SBS). First, a spectrum is interpolated only along Teff, and then it is interpolated along log(g). This method increases an amount of computation increase significantly, but I expected I would get a steady interpolation up to the fourth dimensions (<span class="error">&#91;alpha/Fe&#93;</span>). <br/>  <br/> This is an example of the result with SBS.  <br/> The logg-Teff diagram shows model fluxes (circles) and fluxes of interpolated spectra (squares for the first interpolation along Teff, and a star for the final interpolation) at a fixed wavelength. <br/> The second figure shows the flux residual between interpolations of the SBS method and the RBF method along wavelengths. We can find the residual is significantly larger than that between RBF and a reference model. We cannot expect we get a better result of 3D interpolation with SBS. I switch back to the  RBF interpolation. <br/> <span class="image-wrap" style=""><a id="12109_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12109/12109_RBF3D_Teffd1000_original.png" title="RBF3D_Teffd1000_original.png" file-preview-type="image" file-preview-id="12109" file-preview-title="RBF3D_Teffd1000_original.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12107/_thumb_12107.png" height="185" width="230" style="border: 0px solid black" /></a></span><br/> <span class="image-wrap" style=""><a id="12108_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12108/12108_residual_SBS_RBF.png" title="residual_SBS_RBF.png" file-preview-type="image" file-preview-id="12108" file-preview-title="residual_SBS_RBF.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12108/_thumb_12108.png" height="158" width="427" style="border: 0px solid black" /></a></span><br/>  </p> <h3><a name="3DRBFwithTeff%2F1000"></a><b>3D RBF with Teff/1000</b></h3> <p>I am aware of a problem that the parameter step of Teff is remarkably larger than the others in the current RBF interpolation, although the RBFs are axisymmetric functions. As a test, I divide Teff by 1000 to roughly match its step with those of the others, and then I make an interpolation with 3D RBF (Teff, log(g), Z). The figures are flux residuals between the RBF interpolations and a reference model. The top panel is for a new RBF with the scaled-down Teff and the bottom is for the previous 3D RBF with the original Teff step. We got a better interpolation with a higher accuracy (stddev=0.015%). <br/> <span class="image-wrap" style=""><a id="12109_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/12109/12109_RBF3D_Teffd1000_original.png" title="RBF3D_Teffd1000_original.png" file-preview-type="image" file-preview-id="12109" file-preview-title="RBF3D_Teffd1000_original.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/12109/_thumb_12109.png" height="243" width="328" style="border: 0px solid black" /></a></span><br/>  </p> <p>From <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=takuji.yamashita" class="user-hover" rel="takuji.yamashita">Takuji Yamashita</a>: I decided to use 4D RBF interpolation. This works very well but takes much time (~100min/spectrum). Now I am trying to speed up while I am facing some technical problems. It needs more several days. I will close the ticket after finishing coding it. </p> <p>I pushed codes for this tickets to Git. The codes make an interpolated spectrum with 4D RBF interpolation (`scipy.interpolate.Rbf`) and `multiprocessing` module. The process takes ~60min/spectrum in a machine I use.</p> <p>The bottleneck is a large number of pixels. Current models have a pixel scale of 0.01 nm. Because input templates can be smoothed with Gaussian Kernel of FWHM ~ 0.2 nm, we can use a larger pixel scale of 0.04 nm, for example. In this case, I expect the time is ~ 15min/spectrum.</p> <p>The evaluation of the created interpolation spectra will be done in another ticket.</p> <p>Now the computing time was reduced to ~15min/spectrum using a new high-speed computer we recently purchased. I think it is reasonable.</p> <p>I made two small tickets of <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-460" title="Evaluate the interpolations between model templates" class="issue-link" data-issue-key="PIPE2D-460"><del>PIPE2D-460</del></a> and <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-461" title="Determine parameter resolutions of interpolated AMBRE models" class="issue-link" data-issue-key="PIPE2D-461"><del>PIPE2D-461</del></a> which block this ticket (<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-365" title="Interpolate between model templates " class="issue-link" data-issue-key="PIPE2D-365"><del>PIPE2D-365</del></a>). <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=hassan" class="user-hover" rel="hassan">hassan</a>, could you move these two tickets to the current sprint?</p> <p>Can you clarify whether this 15 min/spectrum is the time to fit the <tt>RBF</tt> to the models and interpolate to a set of parameters, or the time to interpolate to a new set of parameters once the interpolation schema is determined? In other words, how long does it take to interpolate 10 spectra if you do them all at once?</p> <p>The former. </p> <p>The RBF is fitted at a local parameter space, and therefore if I have 10 scattered sets of parameters, it takes 15min * 10; if I have 10 clustered sets (I can use the identical RBF for them), I think the time is shorter (but I have not yet tried that).</p> <p>I am now trying to adjust a number of neighbor models to be used for interpolation. Processing time of the RBF fit is subject to it.</p> <p>Now, the processing time including RBF fit and interpolation is ~3min/spectrum on average. I reduced a number of neighbor spectra for an interpolation from ~250 to ~100 at the small expense of accuracy. The details will be provided in <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-460" title="Evaluate the interpolations between model templates" class="issue-link" data-issue-key="PIPE2D-460"><del>PIPE2D-460</del></a>. I started the process for the evaluation.</p> <p>We were using 4D RBF interpolation. But I realized that we do not need 4D interpolation. We only need 2D, because we do not make small parameter steps along both metallicity and alpha elements. These original steps are small enough (the study of <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-461" title="Determine parameter resolutions of interpolated AMBRE models" class="issue-link" data-issue-key="PIPE2D-461"><del>PIPE2D-461</del></a> ). </p> <p>I changed the code from 4D to 2D (only for temperature and surface gravity). The 2D interpolation is simpler and faster (~9 sec for an interpolation at a grid point). The mass process of the 2D interpolation has been done. Now I am re-calculating the accuracies. </p> <p>I will report a result of the accuracies of 2D RBF in another ticket. Should I make a new ticket for that or use the closed ticket for 4D RBF (<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-460" class="external-link" rel="nofollow">PIPE2D-460</a>)?</p> <p>We can close this ticket to study the parameter interpolation of model templates because we have studied it and evaluated the results in PIEP2D-460. </p> Blocks PIPE2D-461 PIPE2D-511 PIPE2D-364 PIPE2D-460 PIPE2D-510 PIPE2D-537 Development Rank 0|02qpt6:508yl0chzzy0000q01r4 Reviewers hassan Sprint 2DDRP-2019 E 2DDRP-2019 F 2DDRP-2019 G 2DDRP-2019 H 2DDRP-2019 I 2DDRP-2019 J 2DDRP-2019 K 2DDRP-2020 A Story Points 4.0