[PIPE2D-423] Regular DRP processing of LAM and Subaru data Created: 10/May/19  Updated: 13/Jan/22

Status: In Progress
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: hassan Assignee: sogo.mineo
Resolution: Unresolved Votes: 0
Labels: None
Σ Remaining Estimate: Not Specified Remaining Estimate: Not Specified
Σ Time Spent: Not Specified Time Spent: Not Specified
Σ Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relates
relates to PIPE2D-692 Monitor stability data Open
relates to PIPE2D-676 Don't split arms/spectrographs in gen... Done
relates to PIPE2D-677 generateReductionSpec produces detect... Done
relates to PIPE2D-678 Add visit and date restrictions for g... Done
relates to PIPE2D-679 Processing from yaml requires --clean Done
relates to PIPE2D-578 Migrate the integration test to use Y... Done
relates to PIPE2D-600 Migrate the weekly test to use YAML c... Done
relates to PIPE2D-618 Process SM1 stability data Done
relates to INSTRM-1054 Provide logic for processing SM1 cali... Done
Sub-Tasks:
Key
 Summary
 Type
 Status
 Assignee
PIPE2D-565 Write a script to generate a bash scr... Sub-task Done sogo.mineo  
PIPE2D-566 Write a YAML to do the same thing as ... Sub-task Done sogo.mineo  
PIPE2D-639 Restructure YAML inputs to conform to... Sub-task Done sogo.mineo  
PIPE2D-640 Write a script to generate YAML by re... Sub-task Done sogo.mineo  
Story Points: 4
Sprint: 2DDRP-2019 E, 2DDRP-2021 A, 2DDRP-2021 A 2

 Description   

We need to support regular processing of LAM data, where the configuration is specified and persisted in order to track the inputs and reproduce those runs.

A proposal from rhl in an email on 2019-04-03 to Naoki Yasuda is copied below:

Craig and I have a proposal on how to do this in the short term, that can be generalised to work properly with an opDB later.

I'll send around details when I have a chance, but basically we think it's quite easy to define a file (call it foo.yaml) that defines the inputs for all PFS calibrations.  Then a script that can be run in two modes:
	1.  a. Write suitable entries in foo.yaml
	    b. Run the script to generate the specified calibrations from foo.yaml
	    c. Have a human check that the calibs are good (and/or run the quality assessment code)
	    d. Run the script again (with different options) to ingest the new calibrations
	    e. Push foo.yaml to git
or
	2.  a. Pull foo.yaml from git
	    b. Run the script to generate and ingest the calibrations

So we'd carry out the first procedure in one place (LAM?  Princeton?  IPMU?   NAOJ?), then carry out the second everywhere else.

In operations, we'd generate the yaml file from the opDB.

This should be implemented.

 Comments   
Comment by hassan [ 20/May/19 ]

Initial version was pushed to git@github.com:lsst-dm/generateCalibrations.git by rhl on 2019-05-10.

Comment by sogo.mineo [ 07/Aug/20 ]

I am thinking of writing a tool to generate a spec YAML from opDB. What will be the arguments of the tool? Tanaka-san says somebody will already have marked exposures to use, and all the tool will have to do will be find records named "masterbias", "masterflat", etc. If I take this literally, the command line syntax will be

generateSpec.py -d "dbname=opdb host=opdb.example.com"

Is this sufficient? I am afraid it is disregarding chronological changes.

Generated at Sat Feb 10 15:53:00 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.