https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-920 DRP 2-D Pipeline <p>Initially, we plan to store the 6k templates (2.4GB) in GitHub for flux calibration. Because this might be large, Mineo-san and Yamashita are discussing reducing the size.<br/> One option is to upload a small subset of them. This enables us to run a test code for the blue-part in flux calibration but the output is inaccurate. The full data will be uploaded sometime in the future.<br/> Another option is to compress the 6k templates. We are now discussing whether saving them in fixed-point numbers works well or not.<br/>  <br/> We have ~6k templates. In the current plan, a user generates a large template set (5.6k) with smaller parameter grids for stellar typing using the 6k templates and RBF interpolation in the f_star repository, on a user-side. Stellar typing for flux calibration refers to the 5.6k templates. 1 template (1 FITS file) has ~400kB. The size of the first template set of 6k is 6k*400kB = 2.4 GB in total.<br/>  </p> PIPE2D-920

How to put the large set of the AMBRE model templates in github for flux calibration

Story Normal Done Done sogo.mineo Takuji Yamashita flux-calibration model-templates Mon, 25 Oct 2021 07:06:20 +0000 Wed, 22 Dec 2021 00:22:58 +0000 Wed, 22 Dec 2021 00:22:57 +0000 0 6 <p>We lean to saving the 6k templates in fixed-point numbers to reduce the file size. The size could reduce roughly by half, ~1-2GB. We will convert the templates to log and then convert them to fixed-point numbers. We need to test the accuracy.</p> <p>We should also use WCS to do the wavelengths if we can.</p> <p>The spectra are saved in FITS again. We can use WCS for wavelengths. </p> <p>The size estimate appearing in the description does not include wavelength column. FITS files contain flux only. Wavelengths are computed by means of WCS.</p> <p>Missing important comment from @rhl: </p> <blockquote> <p>As we may need to do something cleverer someday, I'd hide it behind an API</p></blockquote> <p>That way, the access and processing software are decoupled from the stored data format.</p> <p>Discussed this a little further with <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=rhl" class="user-hover" rel="rhl">rhl</a> and <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=price" class="user-hover" rel="price">price</a>. Is it possible to store the data on a server somewhere? That would be easier to manage than to store the data under git-lfs. </p> <p>We can indeed put the heavy things in hscdata.mtk.nao.ac.jp, for example. The problem is how to let the test process see them. I would like to hook the first call to <tt>getModelSpectrum()</tt> and get all spectra downloaded into ... some directory. I don't want to use <tt>/tmp</tt> since astropy does this and fills up the limited capacity of <tt>/tmp</tt> all too soon, then killed and leaving the system unstable. Another solution might be to make valid a path starting with <tt>https://</tt> and download the models one by one every time they are requested, just as we actually open and read the model files one by one every time they are requested,</p> <p>I would like to take the last route ("Another solution might be to make valid a path...") because it is the easiest thing to do. If I take this route, it may be that we no longer have to reduce the model size but that we can use all of 60k models, which was Yamashita-san's first plan. One problem is that the task of making calibration references will take a few hours (even if the models are in a local storage) to process a single fiber, so that it will take at least a few hours for a unit test to be proved.</p> <p>I was assuming that the test data would just be a dependency, so it'd be installed once (using as it were <tt>curl</tt>) and then used whenever you run the tests.</p> <p>Then what I have to do are:</p> <ol> <li>Upload the 60k (or smaller 6k) spectra to some http server. (The package might need <tt>ups/</tt> directory)</li> <li>Ask people (I don't know who) to install it, as a dependency, into the server in which automatic tests are run.</li> <li>Push branches that require the package.</li> </ol> <p>Do I understand correctly?</p> <p>Yes, that's great. Please be sure to include a <tt>README</tt> file that explains what the data are and where they came from, and include a version string (usually the date) in the directory name.</p> <p>You're welcome to put it on the tiger cluster at Princeton (<tt>e.g., /projects/HSC/PFS/fluxCal/fluxCal-20211101</tt>), and we can serve it via http from there.</p> <p>I tentatively created a package just now, but I found myself not sure whether or not the synthetic spectra are redistributable. I am now checking it.<br/> The package name will be <tt>fluxmodeldata-ambre-20210512-full</tt> in which <tt>ambre-20210512-full</tt> is its version name.</p> <p>Tanaka-san said we have been given permission of redistribution of the synthetic spectra by the author. Yamashita-san found some flaws in converting the original spectra to the format he uses. He is now recreating the data files.</p> <p>I have uploaded the smaller dataset here <a href="https://hscdata.mtk.nao.ac.jp/hsc_bin_dist/pfs/fluxmodeldata-ambre-20190419-small.tar.xz" class="external-link" rel="nofollow">https://hscdata.mtk.nao.ac.jp/hsc_bin_dist/pfs/fluxmodeldata-ambre-20190419-small.tar.xz</a> .<br/> I intend this package to be used by unit tests because Yamashita-san's algorithm to make flux reference takes several hours per fiber when the full dataset is used.</p> <p>The full dataset has not been completed yet. We found that we had to add more spectra to the dataset, and the spectra have yet to be made. We must also examine whether the tremendous amount of the full dataset and the eon-long execution time really contribute to accuracy of the calibration task.</p> <p>If the above dataset is approved and installed in the server where the tests run, I would like to push changes to <tt>drp_stella</tt> and <tt>drp_pfs_data</tt> that are named this issue. With the changes, the broadband photometry table referred to by <tt>FitBroadbandSEDTask</tt> is moved from <tt>drp_pfs_data</tt> to the above package. The broadband photometry table must reside close to the spectrum set because the two must match with each other.</p> <p>We can close this ticket because we have discussed this issue and Mineo-san has loaded the model template dataset. I will file two new tickets for the works Mineo-san said above:<br/> 1. check and approve the dataset that Mineo-san has uploaded, and then install it on the server.<br/> 2. move the broad-band photometry table to the dataset package<br/> May I assign the 1st one to <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=price" class="user-hover" rel="price">price</a> and the 2nd one to <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=sogo.mineo" class="user-hover" rel="sogo.mineo">sogo.mineo</a> ?</p> <p>I've retrieved the tarball listed above, and placed it in <tt>/projects/HSC/PFS/fluxCal</tt> on our Tiger cluster. It looks good to me.</p> <p>I have made two PR, one of which is to make <tt>drp_stella</tt> depend on <tt>fluxmodeldata</tt> package. The other one is to remove the photometry table from <tt>drp_pfs_data</tt>. The former change should be made before the latter change. Could you review these things?</p> <p>I don't think we can require that every installation of the pipeline contains a 2.4 GB data package. You should make the data package <tt>setupOptional</tt> in the table file, and protect the tests with checks, e.g., <a href="https://github.com/lsst/afw/blob/main/tests/test_maskedImageIO.py#L100" class="external-link" rel="nofollow">here</a>.</p> <p>I changed <tt>fluxmodeldata</tt> from required to optional. Could you review the newly pushed patch? </p> <p>Awesome, thanks!</p> <p>Thanks for the review. I merged my two pull requests to master.</p> Blocks PIPE2D-921 Development Epic Link PIPE2D-303 Rank 0|zzsglb: Reviewers price