[PIPE2D-775] generateCommands should detect bad formats in YAML file Created: 12/Mar/21 Updated: 18/Mar/21 Resolved: 18/Mar/21 |
|
| Status: | Done |
| Project: | DRP 2-D Pipeline |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Story | Priority: | Normal |
| Reporter: | price | Assignee: | sogo.mineo |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Sprint: | 2DDRP-2021 A3 |
| Reviewers: | price |
| Description |
|
Some of the elements of the YAML file can be a bit tricky to get right, especially the bootstrap and fiberProfiles blocks, with their group syntax, e.g.:
bootstrap:
group:
-
flatId: visit=45742
arcId: visit=45744
In such cases, omitting the extra group structure results in generateCommands silently skipping the operation. Please produce errors when the YAML file format will result in incorrect operation, or YAML elements are misnamed or in the wrong place. |
| Comments |
| Comment by sogo.mineo [ 12/Mar/21 ] |
|
Because lsst stack seems to include no library to deal with json schema or the like, and I would have yet to learn such a standard if it were to be available, I am writing check codes directly. |
| Comment by sogo.mineo [ 16/Mar/21 ] |
|
I have made changes. Could you review them? |
| Comment by price [ 17/Mar/21 ] |
|
This looks great! I tried it with the following deliberately misformed YAML file: # To run generateCommands.py with this spec file, # you have to set environment variable DATADIR, # in which initial detectorMap files reside. # # initial detectorMap # # The files are to be found in dirName, taken to be relative to --dataDir if # not an absolute path # init: dirName: "$DATADIR" arms: [b1, r1, n1, m1] detectorMapFmt: "detectorMap-sim-{arm}.fits" # foobar: 12345 # XXX bad name # # Blocks of data which are used together to update the calibrations # calibBlock: - name: calibs_for_brn bias: id: ["field=BIAS", "arm=b^r^n"] # confg: "foo=bar" # XXX misspelling dark: id: ["field=DARK", "arm=b^r^n"] # confgfile: foobar.py # XXX misspelling flat: id: ["field=FLAT", "arm=b^r^n"] # whatever: asdf # XXX bad name # bootstrap: # # XXX missing 'group' # arcId: "visit=39" # flatId: "visit=20" fiberProfiles: # XXX not a list group: id: "visit=35" - name: calibs_for_m fiberProfiles: # XXX missing group - id: ["visit=36", "arm=m"] - id: ["visit=38", "arm=m"] # # Blocks of science pipelines # scienceBlock: # XXX not a list name: pipeline_on_brn id: "visit=47^48^49^53^54^57" This produces an obscure error message: ValueError: dictionary update sequence element #0 has length 1; 2 is required [context: CalibBlock:calibs_for_brn]. Can you add something to clarify that, please? |
| Comment by sogo.mineo [ 17/Mar/21 ] |
|
I force-pushed a change. generateCommands.py now says TypeError: illegal type (must be `list`): `dict` (value: '{'id': 'visit=35'}') [context: CalibBlock:calibs_for_brn] for the spec file above. I would like to ask whether this change is good or not before merging the branch to master, because the change is a little large. |
| Comment by price [ 18/Mar/21 ] |
|
That's wonderful, Mineo-san, thank you! |
| Comment by sogo.mineo [ 18/Mar/21 ] |
|
Thanks for review. Merged to master. |