https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/DAMD-7 Data Model <p>We do not specify how to handle the multiple up the ramp images. I suspect that the answer is MEFs &#8211; but it's not my call &#8211; and we need to specify the details (e.g. what information is in each header).</p> DAMD-7

Define format for up-the-ramp files

Story Major Done Done cloomis rhl Sat, 3 Sep 2016 19:21:29 +0000 Thu, 20 Oct 2022 19:19:18 +0000 Thu, 20 Oct 2022 19:19:18 +0000 0 4 <p>We discussed this at the autumn Princeton Software meeting, and Philip Tait confirmed that MEFs are OK. The next step is generate some example images (possibly from real &#8211; if junky &#8211; H4RG detectors)</p> <p>We are currently saving full raw reads, each in its own RICE-compressed HDU. Note that we do not even do the IRP subtraction, nor IRP de-interleaving. We should at the very least de-interleave the reference columns. [ I propose doing the de-interleaving now, actually. ]</p> <p>In the longer run, once we have cleaner DAQs and preferably some actual 300-fiber spectra, we need to understand how much more efficiently we can compress data by pre-processing the reads (save diffs? etc.)</p> <p>Another question is whether we want to have an "at end" HDU, to save stuff we only learn after starting the ramp.</p> <p>The headers will mostly be identical to the CCD headers. As it stands I am being lazy and saving complete headers for each read. I do not know how well we sample, say, alt/rot positions.</p> <p>Minor file structure issue. We expect to use IRP and de-interlace the data and reference images. I propose that for each read the two be saved in two separate HDUs. If IRP is not used, I would save an empty HDU for the reference image, just to keep the file structure the same. </p> <p>I'm ready to close this ticket, leaving discussions of headers for another place.</p> <p>We save each complete ramp in a PFSB file. For each read in the ramp we save two RICE-compressed uint16 HDUs: one named "IMAGE_%d", the other "REF_%d". The REF header has essentially nothing in it: all per-read metadata goes in the IMAGE header. Other tickets will address just what those cards should be, but the first proposal will likely be for all read-related info (read identification, read timestamps, interleaving details, plus all the environmental and telescope info which changes at interesting timescales).</p> <p>The reference pixel image is raw, in the sense of being only the reference pixels which are read out in the particular ASIC/H4 configuration. So from 0 pixels up to 4k by 4k, depending on the science:reference pixel interleave factor. Any interpolation is for downstream software to do.</p> <p>Could still rename REF_%d and IMAGE_%d....</p> <p>Grr. Was about to close this, I am not able to create empty non-primary image HDUs. How bad would it be to not generate 'REF_%d' HDUs? Since it makes little sense to access HDUs by index not extreme, that seems safe. The reduction code would need to catch missing REF_ HDUs.</p> <p>Will also add an optional RESET_%d HDU, which would come before any IMAGE HDUs, and where %d will almost certainly always be 1. Turns out that I need to read the reset frame, and that there might be value in the content.</p> <p>Could you point me to some instruction of what IRP means and how it operates on these images. I think it acceptable to skip REF HDU in case of disabled IRP, but should have some keyword in the RESET or whatever HDU(s) to indicate which mode was used for readout.</p> <p>"IRP" stands for "Interleaved Reference Pixels", which are extra reads from virtual pixels which tell us something about the readout electronics. You can think of them as being like the overscan pixels in a CCD readout.</p> <p>I'll have to defer to <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a> to tell you about header keywords.</p> <p>To be discussed/clarified in a offline meeting with <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=furu" class="user-hover" rel="furu">Hisanori Furusawa</a>. At working-group meeting level.</p> <p>I was inclined to put the attached into the `ics_hxActor` docs. Maybe it should primarily be in datamodel.</p> <p><tt>PFSB.md</tt> is in <tt>datamodel</tt>. Further work should be on new tickets.</p> Relates DAMD-134 INSTRM-1069 INSTRM-1336 INSTRM-596 INSTRM-1090 Development Rank 0|02qpq3:r3s1r