https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-775 DRP 2-D Pipeline <p>Some of the elements of the YAML file can be a bit tricky to get right, especially the <tt>bootstrap</tt> and <tt>fiberProfiles</tt> blocks, with their <tt>group</tt> syntax, e.g.:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> bootstrap: group: - flatId: visit=45742 arcId: visit=45744 </pre> </div></div> <p>In such cases, omitting the extra <tt>group</tt> structure results in <tt>generateCommands</tt> silently skipping the operation.</p> <p>Please produce errors when the YAML file format will result in incorrect operation, or YAML elements are misnamed or in the wrong place.</p> PIPE2D-775

generateCommands should detect bad formats in YAML file

Story Normal Done Done sogo.mineo price Thu, 11 Mar 2021 21:57:25 +0000 Thu, 18 Mar 2021 00:06:11 +0000 Thu, 18 Mar 2021 00:06:11 +0000 0 2 <p>Because lsst stack seems to include no library to deal with json schema or the like, and I would have yet to learn such a standard if it were to be available, I am writing check codes directly.</p> <p>I have made changes. Could you review them?</p> <p>This looks great! I tried it with the following deliberately misformed YAML file:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # To run generateCommands.py with <span class="code-keyword">this</span> spec file, # you have to set environment variable DATADIR, # in which initial detectorMap files reside. # # initial detectorMap # # The files are to be found in dirName, taken to be relative to --dataDir <span class="code-keyword">if</span> # not an absolute path # init: dirName: <span class="code-quote">"$DATADIR"</span> arms: [b1, r1, n1, m1] detectorMapFmt: <span class="code-quote">"detectorMap-sim-{arm}.fits"</span> # foobar: 12345 # XXX bad name # # Blocks of data which are used together to update the calibrations # calibBlock: - name: calibs_for_brn bias: id: [<span class="code-quote">"field=BIAS"</span>, <span class="code-quote">"arm=b^r^n"</span>] # confg: <span class="code-quote">"foo=bar"</span> # XXX misspelling dark: id: [<span class="code-quote">"field=DARK"</span>, <span class="code-quote">"arm=b^r^n"</span>] # confgfile: foobar.py # XXX misspelling flat: id: [<span class="code-quote">"field=FLAT"</span>, <span class="code-quote">"arm=b^r^n"</span>] # whatever: asdf # XXX bad name # bootstrap: # # XXX missing <span class="code-quote">'group'</span> # arcId: <span class="code-quote">"visit=39"</span> # flatId: <span class="code-quote">"visit=20"</span> fiberProfiles: # XXX not a list group: id: <span class="code-quote">"visit=35"</span> - name: calibs_for_m fiberProfiles: # XXX missing group - id: [<span class="code-quote">"visit=36"</span>, <span class="code-quote">"arm=m"</span>] - id: [<span class="code-quote">"visit=38"</span>, <span class="code-quote">"arm=m"</span>] # # Blocks of science pipelines # scienceBlock: # XXX not a list name: pipeline_on_brn id: <span class="code-quote">"visit=47^48^49^53^54^57"</span> </pre> </div></div> <p>This produces an obscure error message: <tt>ValueError: dictionary update sequence element #0 has length 1; 2 is required [context: CalibBlock:calibs_for_brn]</tt>. Can you add something to clarify that, please?</p> <p>I force-pushed a change. <tt>generateCommands.py</tt> now says</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> TypeError: illegal type (must be `list`): `dict` (value: <span class="code-quote">'{'</span>id<span class="code-quote">': '</span>visit=35<span class="code-quote">'}'</span>) [context: CalibBlock:calibs_for_brn] </pre> </div></div> <p>for the spec file above. I would like to ask whether this change is good or not before merging the branch to master, because the change is a little large.</p> <p>That's wonderful, Mineo-san, thank you!</p> <p>Thanks for review. Merged to master.</p> Development Rank 0|zzs47y:zzzzzzr Reviewers price Sprint 2DDRP-2021 A3