https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/INFRA-4 Software Development Infrastructure <p>I expect that this will be a multi-extension fits file containing data, noise, and flags. We should take a careful look at the SDSS BOSS spectro files and see what we can learn.</p> <p>I expect that the same file format will be used for injection of at least some of the input spectra that the simulator needs. I say, "some" because e.g. arc spectra are probable better described in terms of (lambda, intensity, width) than a 1-D spectrum.</p> INFRA-4

Define format for persistence of 1-D spectra

Story Major Won't Fix Won't Fix rhl rhl Tue, 15 Jul 2014 22:28:03 +0000 Fri, 8 Jul 2016 14:22:52 +0000 Fri, 8 Jul 2016 14:22:52 +0000 0 2 <p>You've already thought about this.</p> <p>The experience I've had so far has been with the SDSS spPlate file format, and I'm leaning toward a modified version of it. The short explanation is that it's multi-extension FITS with an extension for flux, inverse variance (we could use variance instead), sky spectrum subtracted (maybe/maybe-not suitable in our case), bit-flags, and metadata.</p> <p>Here it is described in its modern form for SDSS3:</p> <p><a href="http://data.sdss3.org/datamodel/files/SPECTRO_REDUX/RUN2D/PLATE4/spPlate.html" class="external-link" rel="nofollow">http://data.sdss3.org/datamodel/files/SPECTRO_REDUX/RUN2D/PLATE4/spPlate.html</a></p> <p>And here's the earlier variant from SDSS2:</p> <p><a href="http://classic.sdss.org/dr7/dm/flatFiles/spPlate.html" class="external-link" rel="nofollow">http://classic.sdss.org/dr7/dm/flatFiles/spPlate.html</a></p> <p>For the work I did on spectroscopic variability with Carlos Badenes, we modified the spPlate format to split the spectra into separate files, each containing the co-added spectrum plus the individual spectra which were stacked to produce it. The MEF structure was the same. The result was the so-called spFiber file. I've pinged a few people who've worked with both spFiber files and spPlate files to get some feedback, and by and large the preference is for 1 source per file. The caveat that's been mentioned is that it's often necessary to check neighboring fibers as sources of contamination, and that's easier when all data are in the same file.</p> <p>In bullets, here's why I like the spFiber:</p> <ul> <li>It's based on SDSS, which is: <ul> <li><b>well tested</b>, and</li> <li><b>already familiar</b> to the community.</li> </ul> </li> <li>It's a per-source format, which is: <ul> <li>more easily run in parallel,</li> <li>easier to handle for users who want specific objects.</li> </ul> </li> </ul> <p>One concern I have with it is that there's some metadata which is common to all fibers in the focal plane ... exposure time, observing conditions, etc. In our spFiber files, we just repeated that info in each file, which is clearly inefficient (though not particularly badly so ... it's just not that much info, after all). </p> <p>In any case, I have an example of an spFiber, and a short Python script which loads it to make a trivial plot of the coadd and its sub-spectra. Where is a suitable place to check these in, or to otherwise share them?</p> <p>Also, although I'm leaning towards the spFiber model, I don't consider it a done deal. If others have suggestions/ideas/concerns, please speak.</p> <p>Let me channel a pair of questions which I have been asked about using SDSS reductions. The proposed per-fiber model (which I like) opens some choices up.</p> <ul class="alternate" type="square"> <li>Do we provide pre-coadd "spFiber"s, where pixel wavelengths have <b>not</b> been interpolated onto a common vector? I would expect 3 rows or 3 HDUs for the three cameras. Basically: can people get "well-calibrated" detector pixels with unambiguous bitmasks?</li> </ul> <ul class="alternate" type="square"> <li>For the coadd spFibers, do we lock the wavelength CRVAL1 to be int*CD1_1, where CD1_1 is a constant for all reductions? Even for coadds of one exposure? Basically: can spFibers be directly compared/grouped without interpolation, which process no one gets right in the face of complex bitmasks?</li> </ul> <p>This work has been moved into the data model project (DAMD), and is tracked in git at <a href="https://github.com/Subaru-PFS/datamodel/blob/master/datamodel.txt" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/datamodel/blob/master/datamodel.txt</a></p> Blocks SIM2D-28 SIM2D-29 SIM2D-31 SIM2D-32 SIM2D-33 Relates SIM2D-35 Development Epic Link DAMD-14 Rank 0|02qpud:r Reviewers cloomis Sprint 2014-12 2014-13 2014-14