[PIPE2D-656] Generate/update headerFix files based on opDb spsAnnotation.data_flag

Relates — Fri, 6 Jan 2023 13:01:09 +0000

We have, I believe, agreed to use the opDb sps_annotation table to record bad PFS raw data files. Basically, if there is a row for the (visit, cam) and the data_flag column is not the "OK" value, the data should not be reduced.

Mineo-san's scripts already use the sps_annotation table to drive reductions. This change would apply to files which are not processed by that engine.

DRP uses a "headerFix" mechanism to track edits to header cards, which are applied when the file is read in: the raw data files are not themselves edited. This could be used to add a card indicating that the data are bad, in which case most of the machinery is already in place.

So:

agree on how/when opDb data is propagated
settle on a data quality card for the DRP.
arrange for ingest or the first isr stage to skip bad files
write opDb -> headerFix script

This is probably just an RFC ticket, until the right people check off on it...

PFS-JIRA

[PIPE2D-656] Generate/update headerFix files based on opDb spsAnnotation.data_flag