-
Type: Story
-
Status: Done (View Workflow)
-
Priority: Normal
-
Resolution: Done
-
Labels:None
-
Sprint:Gen3 transition
We need a facility to allow us to track corrections to and problems with individual spectra. We want to be able to store values (e.g., blackspot correction of 10%) in addition to flags (e.g., aperture correction failed). This would be a good place to put QA metrics measured during pipeline operation too.
The desire to have key-value pairs where the values are of varying types (bool, int, float, string) makes it difficult/inconvenient to represent in FITS using conventional means. However, we would like to avoid using something python-specific like pickle, so that the data is easily accessible from any programming language. I think that the proposed representation, while perhaps unconventional, is simple and meets these design requirements.
I propose to add a new HDU to spectral products (pfsArm, pfsMerged, pfsSingle, pfsObject). The reduction notes (NOTES) HDU is a FITS image HDU consisting of a UTF-8-encoded JSON representation of (for multi-spectra products, an array of NFIBER collections of) key-value pairs that record operations performed and measurements made during reduction. The HDU may be compressed using standard FITS compression conventions.
I have a proposed implementation created in the course of working on PIPE2D-1143, and welcome comments and suggestions.
- blocks
-
PIPE2D-1143 sky subtraction is bad for a certain fraction of fibers in the faint galaxy frames taken during Nov run
- Done