[PIPE2D-775] generateCommands should detect bad formats in YAML file Created: 12/Mar/21  Updated: 18/Mar/21  Resolved: 18/Mar/21

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Normal
Reporter: price Assignee: sogo.mineo
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Sprint: 2DDRP-2021 A3
Reviewers: price

 Description   

Some of the elements of the YAML file can be a bit tricky to get right, especially the bootstrap and fiberProfiles blocks, with their group syntax, e.g.:

    bootstrap:
      group:
        -
          flatId: visit=45742
          arcId: visit=45744

In such cases, omitting the extra group structure results in generateCommands silently skipping the operation.

Please produce errors when the YAML file format will result in incorrect operation, or YAML elements are misnamed or in the wrong place.



 Comments   
Comment by sogo.mineo [ 12/Mar/21 ]

Because lsst stack seems to include no library to deal with json schema or the like, and I would have yet to learn such a standard if it were to be available, I am writing check codes directly.

Comment by sogo.mineo [ 16/Mar/21 ]

I have made changes. Could you review them?

Comment by price [ 17/Mar/21 ]

This looks great! I tried it with the following deliberately misformed YAML file:

# To run generateCommands.py with this spec file,
# you have to set environment variable DATADIR,
# in which initial detectorMap files reside.

#
# initial detectorMap
#
# The files are to be found in dirName, taken to be relative to --dataDir if
# not an absolute path
#
init:
  dirName: "$DATADIR"
  arms: [b1, r1, n1, m1]
  detectorMapFmt: "detectorMap-sim-{arm}.fits"
#  foobar: 12345  # XXX bad name

#
# Blocks of data which are used together to update the calibrations
#
calibBlock:
  -
    name: calibs_for_brn
    bias:
      id: ["field=BIAS", "arm=b^r^n"]
#      confg: "foo=bar"  # XXX misspelling
    dark:
      id: ["field=DARK", "arm=b^r^n"]
#      confgfile: foobar.py  # XXX misspelling
    flat:
      id: ["field=FLAT", "arm=b^r^n"]
#      whatever: asdf  # XXX bad name
#    bootstrap:
#      # XXX missing 'group'
#      arcId: "visit=39"
#      flatId: "visit=20"
    fiberProfiles:
      # XXX not a list
      group:
        id: "visit=35"
  -
    name: calibs_for_m
    fiberProfiles:
      # XXX missing group
      -
        id: ["visit=36", "arm=m"]
      -
        id: ["visit=38", "arm=m"]

#
# Blocks of science pipelines
#
scienceBlock:
  # XXX not a list
  name: pipeline_on_brn
  id: "visit=47^48^49^53^54^57"

This produces an obscure error message: ValueError: dictionary update sequence element #0 has length 1; 2 is required [context: CalibBlock:calibs_for_brn]. Can you add something to clarify that, please?

Comment by sogo.mineo [ 17/Mar/21 ]

I force-pushed a change. generateCommands.py now says

TypeError: illegal type (must be `list`): `dict` (value: '{'id': 'visit=35'}') [context: CalibBlock:calibs_for_brn]

for the spec file above. I would like to ask whether this change is good or not before merging the branch to master, because the change is a little large.

Comment by price [ 18/Mar/21 ]

That's wonderful, Mineo-san, thank you!

Comment by sogo.mineo [ 18/Mar/21 ]

Thanks for review. Merged to master.

Generated at Sat Feb 10 15:57:35 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.