https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-565 DRP 2-D Pipeline <p>Write a python script that generates from a yaml a bash script to create calibs and run science pipeline.</p> PIPE2D-565

Write a script to generate a bash script from yaml

Sub-task PIPE2D-423 Normal Done Done sogo.mineo sogo.mineo Thu, 23 Apr 2020 06:36:27 +0000 Thu, 13 Jan 2022 14:35:36 +0000 Thu, 4 Jun 2020 00:34:15 +0000 0 3 <p>Wrote one. (But I don't know where to push it)</p> <p>Can you initially add the python script to this ticket please?</p> <p>I uploaded the python script forked from <a href="https://github.com/lsst-dm/generateCalibrations" class="external-link" rel="nofollow">https://github.com/lsst-dm/generateCalibrations</a></p> <p>Added the code to <tt>pfs_pipe2d</tt> and pushed branch <tt>tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-565" title="Write a script to generate a bash script from yaml" class="issue-link" data-issue-key="PIPE2D-565"><del>PIPE2D-565</del></a></tt>.</p> <p>I think this needs some more work to achieve what we need. This could supplant <tt>pfs_build_calibs.sh</tt>, but it's missing some important features.<br/> I think it's important to update <tt>pfs_integration_test.sh</tt> to use this method (or provide a similar script) so that we can check that this works, and exercise it regularly.<br/> The code needs to be edited for style (make sure it passes <tt>flake8</tt> with the standard modifications) and docstrings added.</p> <p>I think I have made all required changes, except for adding capability of specifying more keys for --id.<br/> What keys can be used? I thought I could refer to the schema of table <tt>raw</tt> (below), but apparently those columns of <tt>double</tt> type cannot be used for --id.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> CREATE TABLE raw ( id integer primary key autoincrement, site text, category text, field text, visit <span class="code-object">int</span>, ccd <span class="code-object">int</span>, filter text, arm text, spectrograph <span class="code-object">int</span>, dateObs text, expTime <span class="code-object">double</span>, dataType text, taiObs text, pfsDesignId <span class="code-object">int</span>, slitOffset <span class="code-object">double</span>, dither <span class="code-object">double</span>, shift <span class="code-object">double</span>, focus <span class="code-object">double</span>, lamps text, attenuator <span class="code-object">double</span>, photodiode <span class="code-object">double</span>, ) </pre> </div></div> <p>Even the <tt>double</tt> fields can be used in the <tt>&#45;&#45;id</tt>, e.g., <tt>&#45;&#45;id field=FLAT exptime=10.0</tt>.</p> <p>Thank you. I made all fields available in the YAML spec file.</p> <p>I'm not sure hard-coding all the available keys is the right thing to do. The command-line syntax is free-form, and I think that's the way to go here too. For example:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> id: field=FLAT exptime=10.0 </pre> </div></div> <p>Are you concerned about spelling mistakes?</p> <p>I see both advantages and disadvantages to your chosen implementation. Can you explain your reasoning, please?</p> <p>I wanted to check the syntax of <tt>"value"</tt> in <tt>"--id key=value"</tt>, needed to know the type of <tt>"value"</tt> in advance, and enumerated all <tt>key</tt> s with their types. But this is not essential. And I agree that writing <tt>"id: <span class="error">&#91;key=value, key=value&#93;</span>"</tt> is clearer than mixing these <tt>"key=value"</tt> with other things like <tt>"config:..."</tt> and <tt>"validity:..."</tt>. I am revising the code.</p> <p>Currently, the output shell script is not a self-contained executable script but a mere function body. One has to source it (<tt>source out.sh</tt>) in order to execute it. Now that the shell script is written to a file, I think the shell script should be a self-contained executable script that begins with <tt>#!/bin/sh</tt>. Is it OK to make this change?</p> <p><a href="https://developer.lsst.io/work/flow.html#git-commit-organization-best-practices" class="external-link" rel="nofollow">https://developer.lsst.io/work/flow.html#git-commit-organization-best-practices</a> seems to say that I have to squash all commits made after the first review, rebase <tt>tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-565" title="Write a script to generate a bash script from yaml" class="issue-link" data-issue-key="PIPE2D-565"><del>PIPE2D-565</del></a></tt> onto the current <tt>master</tt> branch, and force-push the rebased <tt>tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-565" title="Write a script to generate a bash script from yaml" class="issue-link" data-issue-key="PIPE2D-565"><del>PIPE2D-565</del></a></tt> to github. Do I correctly understand the document?</p> <p>Most important is that there is one functional change per commit. Ideally, there should be a one-to-one relationship between functional changes and commits (e.g., changes to the same code made before and after review get squashed together), but sometimes that is difficult to achieve, and so there may be multiple commits to achieve one overall functional change. But there should not be two functional changes in the same commit if you can at all help it. Does that make sense?</p> <p>I think having the script be executable rather than sourced is a good idea.</p> <p>I changed my code so that it will produce an executable script, and also added bootstrap functionality just now. Though I said in slack that I would provide <tt>lineList</tt> field for <tt>bootstrap</tt>, I decided not to provide it because it was not provided for <tt>reduceArc.py</tt> either.</p> <p>I believe the overall code is ready for re-review. Could you check the new code? (I'm sorry for its big change from what it was on the first review)</p> <p>I changed a few lines in order to promote <tt>--output</tt> parameter to a positional parameter, and to replace <tt>logger.info(f"message {s}")</tt> to <tt>logger.info("message %s", s)</tt>. I squashed this change onto the existing commit and pushed it. I wrongly thought github would still be able to show diffs, but what I see now is only the new code of 1500 lines with no history. I should not have done this...</p> <p>Ah, I found how to view diffs. Sorry for making a noise.</p> <p>I learned just now that it is I who have to merge this ticket branch to master. The latest code (as I have pushed to github) is a little different from what Paul approved, but am I still allowed to merge this branch?</p> <p>Please go ahead.</p> <p>Thank you. I have merged this ticket branch to master.</p> Relates PIPE2D-578 Development Rank 0|02qpq3:r0a Reviewers price Sprint 2DDRP-2019 E 2DDRP-2021 A 2DDRP-2021 A 2