https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/DAMD-145 Data Model <p>We need a facility to allow us to track corrections to and problems with individual spectra. We want to be able to store values (e.g., blackspot correction of 10%) in addition to flags (e.g., aperture correction failed). This would be a good place to put QA metrics measured during pipeline operation too.</p> <p>The desire to have key-value pairs where the values are of varying types (bool, int, float, string) makes it difficult/inconvenient to represent in FITS using conventional means. However, we would like to avoid using something python-specific like pickle, so that the data is easily accessible from any programming language. I think that the proposed representation, while perhaps unconventional, is simple and meets these design requirements.</p> <p>I propose to add a new HDU to spectral products (pfsArm, pfsMerged, pfsSingle, pfsObject). The reduction notes (<tt>NOTES</tt>) HDU is a FITS image HDU consisting of a UTF-8-encoded JSON representation of (for multi-spectra products, an array of <tt>NFIBER</tt> collections of) key-value pairs that record operations performed and measurements made during reduction. The HDU may be compressed using standard FITS compression conventions.</p> <p>I have <a href="https://github.com/Subaru-PFS/datamodel/compare/tickets/PIPE2D-1143" class="external-link" rel="nofollow">a proposed implementation</a> created in the course of working on <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1143" title="sky subtraction is bad for a certain fraction of fibers in the faint galaxy frames taken during Nov run" class="issue-link" data-issue-key="PIPE2D-1143"><del>PIPE2D-1143</del></a>, and welcome comments and suggestions.</p> DAMD-145

Add facility to hold history/metrics of spectra

Story Normal Done Done price price Sat, 21 Jan 2023 00:35:28 +0000 Thu, 16 Feb 2023 16:10:08 +0000 Thu, 16 Feb 2023 16:10:08 +0000 0 8 <p>sounds good to me, are such comments readable/usable by the subsequent modules of the pipeline (eg PIPE1D), if we want to avoid errors which would be hard to detect in the spectrum itself but easy to anticipate from those comments, or will they be redundant to the flags anyway ?</p> <p>Here's an alternative proposal that avoids JSON:</p> <p>The reduction notes (<tt>NOTES</tt>) HDU is a FITS table HDU consisting of the following columns:</p> <ul> <li><tt>name</tt> (string): the keyword name</li> <li><tt>blank</tt> (variable-length array of int): indices of spectra for which the keyword is undefined.</li> <li><tt>bool</tt> (variable-length array of bool): values for spectra, if the field is of boolean type; otherwise, empty.</li> <li><tt>int</tt> (variable-length array of int): values for spectra, if the field is of integer type; otherwise, empty.</li> <li><tt>float</tt> (variable-length array of float): values for spectra, if the field is of floating-point type; otherwise, empty.</li> <li><tt>string</tt> (variable-length array of string): values for spectra (<tt>NUL</tt>-separated), if the field is of string type; otherwise empty.</li> </ul> <p>It's a fair bit more complicated, but it might save space by not having to write the keyword name many times and writing numbers as strings.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=vlebrun" class="user-hover" rel="vlebrun">vlebrun</a>: the <tt>notes</tt> will be implemented in the python <tt>PfsObject</tt> class that 1D operates on. It describes the spectrum as a whole, whereas the mask ("flags") describe individual pixels within the spectrum. It may contain indications that the spectrum is not of good quality, but 1D should probably process it anyway and allow the user to choose whether to include it in their analysis or not.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=price" class="user-hover" rel="price">price</a>: If I understand correctly your alternative proposal, for a given file, the size of the "variable-length" arrays is NFIBER, isn't it ?</p> <p>The number of entries in <tt>blank</tt> plus the number of entries in one of <tt>bool</tt>, <tt>int</tt>, <tt>float</tt> or <tt>string</tt> should be <tt>NFIBER</tt>.</p> <p>I think that will be very useful for QA as well. I like JSON indeed, if permitted. I'm not sure if there is any restriction or policy on FITS when NAOJ delivers the output to users. And also, this is somewhat related to QA, so I take the liberty of inviting <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=msyktnk" class="user-hover" rel="msyktnk">Masayuki Tanaka</a> to this discussion.</p> <p>I think I prefer the alternative proposal (FITS table) to the original (JSON). It's a bit more complicated, but it's probably more compact and doesn't require parsing text.</p> <p>Adding an HDU for this sort of thing is a good idea, which I endorse. I also agree that pickle is a non-starter.</p> <p>However, I'm not sure that I like a free-form non-datamodel controlled HDU. How will we use this? I'd rather provide a light-weight way to add more keys to this HDU, or more precisely the python object which it maps to. Do you have a sketch of this object?</p> <p>As part of this, how would we manage schema evolution? &#8211; I think we only need to worry about adding more fields.</p> <p>Not sure what sort of QA we are thinking of here, but I had a bit of discussion with Hamano-san yesterday.  Hamano-san made a good point; it makes a lot of sense to include QA metrics from the pipeline in the fits header (e.g., number/area of CR), but as we agreed before, QAs that NAOJ/IPMU are working on is a post-processing task, and we would prefer not to update the fits header</p> <p>The FITS header doesn't make sense for QA about individual spectra, because there are 650 spectra per spectrograph. It needs to go in a separate HDU.</p> <p>Sorry, I meant to say HDU instead of fits header.  It might be OK to include detector-wide QA in the header, though.</p> <p>Koike-san is afraid of JSON being unable to distinguish between integers and floats. Many 64bit integers like <tt>pfsDesignId</tt> exceeds 53bits. If they appear in the saved data, the stringified JSON won't be compatible though python's <tt>json</tt> module can deal with them correctly. I add <tt>Infinity</tt> and <tt>NaN</tt> to him.</p> <p>I'm going to take this as generally approved, with the amendments:</p> <ul> <li>The alternative proposal will be used for persistence.</li> <li>The implementation will enforce a controlled schema.</li> </ul> <p>I'll change the implementation on <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1143" title="sky subtraction is bad for a certain fraction of fibers in the faint galaxy frames taken during Nov run" class="issue-link" data-issue-key="PIPE2D-1143"><del>PIPE2D-1143</del></a>, and close this once it's done.</p> <p>There's a new implementation on <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1143" title="sky subtraction is bad for a certain fraction of fibers in the faint galaxy frames taken during Nov run" class="issue-link" data-issue-key="PIPE2D-1143"><del>PIPE2D-1143</del></a>.</p> <p><a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-1143" title="sky subtraction is bad for a certain fraction of fibers in the faint galaxy frames taken during Nov run" class="issue-link" data-issue-key="PIPE2D-1143"><del>PIPE2D-1143</del></a> merged to master.</p> Blocks PIPE2D-1143 Development Rank 0|02qpjd:00r200603w Sprint 2DDRP-2023 A