Uploaded image for project: 'Data Model'
  1. Data Model
  2. DAMD-145

Add facility to hold history/metrics of spectra

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done (View Workflow)
    • Priority: Normal
    • Resolution: Done
    • Labels:
      None
    • Sprint:
      Gen3 transition

      Description

      We need a facility to allow us to track corrections to and problems with individual spectra. We want to be able to store values (e.g., blackspot correction of 10%) in addition to flags (e.g., aperture correction failed). This would be a good place to put QA metrics measured during pipeline operation too.

      The desire to have key-value pairs where the values are of varying types (bool, int, float, string) makes it difficult/inconvenient to represent in FITS using conventional means. However, we would like to avoid using something python-specific like pickle, so that the data is easily accessible from any programming language. I think that the proposed representation, while perhaps unconventional, is simple and meets these design requirements.

      I propose to add a new HDU to spectral products (pfsArm, pfsMerged, pfsSingle, pfsObject). The reduction notes (NOTES) HDU is a FITS image HDU consisting of a UTF-8-encoded JSON representation of (for multi-spectra products, an array of NFIBER collections of) key-value pairs that record operations performed and measurements made during reduction. The HDU may be compressed using standard FITS compression conventions.

      I have a proposed implementation created in the course of working on PIPE2D-1143, and welcome comments and suggestions.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                price price
                Reporter:
                price price
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: