[PIPE2D-656] Generate/update headerFix files based on opDb spsAnnotation.data_flag Created: 21/Nov/20  Updated: 06/Jan/23

Status: Open
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Normal
Reporter: cloomis Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: opDB
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates
relates to INSTRM-1202 Add command to create sps_annotations Done

 Description   

We have, I believe, agreed to use the opDb sps_annotation table to record bad PFS raw data files. Basically, if there is a row for the (visit, cam) and the data_flag column is not the "OK" value, the data should not be reduced.

Mineo-san's scripts already use the sps_annotation table to drive reductions. This change would apply to files which are not processed by that engine.

DRP uses a "headerFix" mechanism to track edits to header cards, which are applied when the file is read in: the raw data files are not themselves edited. This could be used to add a card indicating that the data are bad, in which case most of the machinery is already in place.

So:

  • agree on how/when opDb data is propagated
  • settle on a data quality card for the DRP.
  • arrange for ingest or the first isr stage to skip bad files
  • write opDb -> headerFix script

This is probably just an RFC ticket, until the right people check off on it...


Generated at Sat Feb 10 15:56:06 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.