Uploaded image for project: 'DRP 2-D Pipeline'
  1. DRP 2-D Pipeline
  2. PIPE2D-656

Generate/update headerFix files based on opDb spsAnnotation.data_flag

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Open (View Workflow)
    • Priority: Normal
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      We have, I believe, agreed to use the opDb sps_annotation table to record bad PFS raw data files. Basically, if there is a row for the (visit, cam) and the data_flag column is not the "OK" value, the data should not be reduced.

      Mineo-san's scripts already use the sps_annotation table to drive reductions. This change would apply to files which are not processed by that engine.

      DRP uses a "headerFix" mechanism to track edits to header cards, which are applied when the file is read in: the raw data files are not themselves edited. This could be used to add a card indicating that the data are bad, in which case most of the machinery is already in place.

      So:

      • agree on how/when opDb data is propagated
      • settle on a data quality card for the DRP.
      • arrange for ingest or the first isr stage to skip bad files
      • write opDb -> headerFix script

      This is probably just an RFC ticket, until the right people check off on it...

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                cloomis cloomis
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: