-
Type: Story
-
Status: In Progress (View Workflow)
-
Priority: Normal
-
Resolution: Unresolved
-
Affects Version/s: None
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Story Points:4
-
Sprint:2DDRP-2019 E, 2DDRP-2021 A, 2DDRP-2021 A 2
We need to support regular processing of LAM data, where the configuration is specified and persisted in order to track the inputs and reproduce those runs.
A proposal from rhl in an email on 2019-04-03 to Naoki Yasuda is copied below:
Craig and I have a proposal on how to do this in the short term, that can be generalised to work properly with an opDB later. I'll send around details when I have a chance, but basically we think it's quite easy to define a file (call it foo.yaml) that defines the inputs for all PFS calibrations. Then a script that can be run in two modes: 1. a. Write suitable entries in foo.yaml b. Run the script to generate the specified calibrations from foo.yaml c. Have a human check that the calibs are good (and/or run the quality assessment code) d. Run the script again (with different options) to ingest the new calibrations e. Push foo.yaml to git or 2. a. Pull foo.yaml from git b. Run the script to generate and ingest the calibrations So we'd carry out the first procedure in one place (LAM? Princeton? IPMU? NAOJ?), then carry out the second everywhere else. In operations, we'd generate the yaml file from the opDB.
This should be implemented.
- relates to
-
PIPE2D-692 Monitor stability data
- Open
-
PIPE2D-676 Don't split arms/spectrographs in generateReductionSpec
- Done
-
PIPE2D-677 generateReductionSpec produces detectorMap blocks with hundreds of visits
- Done
-
PIPE2D-678 Add visit and date restrictions for generateReductionSpec
- Done
-
PIPE2D-679 Processing from yaml requires --clean
- Done
-
PIPE2D-578 Migrate the integration test to use YAML config files
- Done
-
PIPE2D-600 Migrate the weekly test to use YAML config files
- Done
-
PIPE2D-618 Process SM1 stability data
- Done
-
INSTRM-1054 Provide logic for processing SM1 calibration data
- Done