https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/INSTRM-354 Instrument control development <p>As it stands, when the routines which generate FITS cards get an invalid or missing value, they do not generate the card, but write a COMMENT instead.</p> <p>We should write out 'NaN' for floating point numbers and, say, 'INVALID' for integers. Probably leave the semi-informative COMMENTS, but it is better for readers to get invalid values than no card.</p> <p>n.b. NaNs are valid pixel values, but NaN is not a valid card value.</p> INSTRM-354

On missing, unconverted, or invalid numeric values, write a string FITS complaint

Task Normal Done Done cloomis cloomis FITS Wed, 2 May 2018 19:07:45 +0000 Fri, 22 Apr 2022 05:43:41 +0000 Fri, 22 Apr 2022 05:43:41 +0000 0 4 <p>Pinning the conventions down a bit more:</p> <p>Unavailable values (where there is no Gen2 or MHS keyword at all, say):</p> <ul class="alternate" type="square"> <li>Subaru already uses <tt>'##NODATA##'</tt>. Can we always use that, as ugly as it is? Good for all types.</li> </ul> <p>Invalid and out-of-range values:</p> <ul class="alternate" type="square"> <li><tt>FLTCARD = NaN</tt> is not valid FITS: use <tt>FLTCARD = 'NaN'</tt>.</li> <li>Because of that, should we simply use <tt>INTCARD = 'NaN'</tt>? Using a string which suggests a float might be a problem.</li> </ul> <p>The most common use-case for the invalid integer is for stepper motors whose position is not known. Yes, you can often use <tt>-9999</tt> etc., but that often causes trouble.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=furu" class="user-hover" rel="furu">Hisanori Furusawa</a> check how invalid values are converted prior to ingestion. By the next FITS WG meeting.</p> <p>A meta keyword that qualifies (valid/invalid) the keyword in question may be an option.</p> <p>Per the 4.0 standard's 4.1.2.3:</p> <p>"The value field, when present, shall contain the ASCII text representation of a literal string constant, a logical constant, or a numerical constant, in the format specified in Sect. 4.2. The value field may be a null field; i.e., it may consist entirely of spaces, in which case the value associated with the keyword is undefined."</p> <p>So we could just leave out the value. Is that OK for Subaru?</p> <p>We are checking this - sorry for taking long, but could you elaborate on what exactly the problems are if we set the values like -9999? I am still inclined to set these normal but impossible numbers as default, since they are simply db loadable as they are without any type conversion, and will not conflict with real measurements. We could define multiple values for different states like -9999 (measurement failure), -8888 (not set; not ready etc for known reason)..</p> <p>If type conversion is a significant problem for the db ingest programs, I agree that using special values is probably the right choice. I was hoping to use a defined FITS standard, but if we can't so be it.<br/> Do you know whether the ingest programs handle <em>any</em> special values, like the string "NaN" for a floating point value? I also wonder whether the common Gen2 keyword value "##NODATA##" is handled specially?</p> <p>Let me talk to myself out loud.</p> <p>The source of every FITS card is an MHS keyword. Each MHS keyword has a special invalid value, which is generally unused or which defaults to NaN. That value is currently used when sending MHS keywords <em>within</em> the ICS system.<br/> We could use that mechanism, certainly for strings and integers, and use some defaults (your suggested -9999 or -8888, say)</p> <p>Thinking through our devices, I suspect that -9999, etc. would work. Sigh.</p> <p>We do need to decide on this very soon, since people are starting to use the data files.</p> <p>I'll be generating two special values, "invalid" and "out-of-date". The real NaNs, etc. can in general be useful, so those are still used within the instrument software. The new special values are <b>only</b> used for FITS headers. We need cover the four header card types: integer, real, string, and logical.</p> <p>For invalid, -9999, -9999.0, and "invalid value". The comment will have "INVALID" appended.<br/> For out-of-date, -9998, -9998.0, and "expired value". The comment will have "NOT CURRENT" appended.</p> <p>Logicals are left untouched in the case of expired values, and have no invalid value.</p> <p>Am adding the "value" to the strings so as not to get confused with the common "invalid" enumeration value (both limit switches being set, for instance).</p> <p>Some measurement errors will show other "invalid" values. For instance the cryostat temperatures will be -9999 for no readings, 400K for off-scale high, and 0K for off scale low. In this case, the internal NaN signals that an ADC channel could not be read, and the other two that the circuit is open/shorted. This ticket only translates the NaN.</p> <p>If we need a different special value for some cards, it would not be hard to add the mechanism. But I am not doing it until we actually encounter the problem.</p> <p>Can you check off on this please, <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=furu" class="user-hover" rel="furu">Hisanori Furusawa</a>?</p> <p>Thank you for your proposal. Could you elaborate a bit on it? Do you mean with the "invalid value" and "expired value" string values will be set in the FITS header, or they are still numeric that is the same type as regular values for a certain card? For instance, are the invalid values 400 and 0 for the cryostat temperatures?<br/> I believe a concern by the archive team is that if the value type in a certain card varies frame by frame, they need to deal with it in their ingesting program, which leads to cumbersome case-by-case exceptional code. So, we want to ingest values as they are without any type casting or pre-processing for conversion for reliable operation and maintenability of the code.</p> <p>Sorry, I meant that for when the FITS writer cannot determine a valid value (the internal value is "NaN" or None or blank, or "##NODATA##"), it will use a <b>type-correct</b> value: -9999 for integer cards, -9999.0 for float cards, and "invalid value" for string cards. Logical cards will always be T or F.</p> <p>Likewise for "expired" values. In some sense there is no difference between "invalid" and "expired" values, but I would like to be able to express as much as I can about things which have gone wrong. But if you would prefer I can convert all failures to the "invalid" values.</p> <p>I only mentioned the cryostat temperatures to point out that there may be other type-correct values which indicate errors.</p> <p>Thank you. I understand. This plan looks fine to me. They want to have a dictionary that defines such special values for each card, so that they could perform validation of data on the archive side. <br/> Currently, SMOKA ingests '##NODATA##' without any conversion since it always appears in string cards and there is no conversion issue, but I agree on that, instead, having dedicated values for failures such as "invalid value" and "expired value" is good for health check and data quality control.</p> <p>Long since done.</p> Relates INSTRM-484 PIPE2D-404 INSTRM-130 Development Rank 0|ii04hr: