https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-405 DRP 2-D Pipeline <p>I'd like to propose a minimal header patching scheme for comments/review. Unpolished but functional. Here's a doc and a test YAML file. </p> PIPE2D-405

RFC: header patching

Task Normal Done Done price cloomis Mon, 15 Apr 2019 14:26:58 +0000 Wed, 25 Sep 2019 19:12:53 +0000 Wed, 25 Sep 2019 18:46:41 +0000 0 4 <p>Is there a sane place in the butler to put a function which modifies a header? The test code takes a fits pathname and returns a pyfits header &#8211; not sure whether such a thing can be used. <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=price" class="user-hover" rel="price">price</a>?</p> <p>Try <tt>PfsMapper.std_raw</tt> and <tt>std_raw_md</tt>. See <a href="https://github.com/lsst/obs_subaru/blob/master/python/lsst/obs/hsc/hscMapper.py#L166-L188" class="external-link" rel="nofollow">HscMapper</a> for examples.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=price" class="user-hover" rel="price">price</a> when you have the opportunity, can you review the proposal please?</p> <p>Looks reasonable to me.<br/> I suggest running it by Tim Jenness (LSST), as he's thought about these kinds of things.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a> to add initial code to git and provide link in this ticket.</p> <p>This has not changed since the tickets was opened. For evaluation, am just providing an executable, not tests, sorry. From <tt>$OBS_PFS_DIR</tt>, <tt>python/lsst/obs/pfs/patchHeader.py --yamlDir=tests/patches /your/path/to/PFLA01499...</tt> will show what would happen.</p> <p>This is assigned for me, and it's not clear what I'm expected to do. I'm thinking I'm to implement this scheme in the butler?</p> <p>On a second reading, I have the following comments:</p> <p>&gt; We propose saving YAML files containing a dictionaries of list of rules, in the pfs_instdata git repo.</p> <p>The pfs_instdata repo is HUGE, and having to download it would compromise our continuous integration with Travis (which i think must set time limits on jobs). Can we put these in their own repo instead?</p> <p>I'm not necessarily opposed to YAML files with rules, but I wonder if it might not be easier to use python code instead, with a number of standard functions defined that do the various adding/modifying/deleting of keywords?</p> <p>&gt; Are filename globs the right key?</p> <p>Not sure. When we're using this in the butler, we have keyword-value pairs like <tt>visit=123 arm=r spectrograph=4</tt>. We can construct a raw filename, but that uses the <tt>PF*r1.fits</tt>-like style of filenames rather than <tt>PF*12.fits</tt> style. Of course, we can convert back to the latter style; it's just annoying.</p> <p>Since this was opened Tim Jenness has implemented a viable header patching scheme for LSST, and we should probably adopt it instead.</p> <p> </p> <p>Tim's header patching scheme is implemented in the <a href="https://github.com/LSST/astro_metadata_translator" class="external-link" rel="nofollow">astro_metadata_translator</a> package, which is intended to be a general-purpose astronomy metadata translator (FITS headers to astronomical metadata), and is used by the butler to generate <tt>VisitInfo</tt> from the FITS headers. There is a <tt>fix_header</tt> function that searches for a YAML file (specified by observation ID) that sets values in the header (it has no ability to set values programmatically); this is used within the butler. The path for the YAML files can be specified explicitly, or via a envvar (<tt>METADATA_CORRECTIONS_PATH</tt>).</p> <p>I <em>think</em> this will be sufficient for our purposes if we add a layer over the top to generate the YAML files programmatically. Tim is concerned that the YAML files be available for people to download without needing a bunch of prerequisites, so I'm hopeful that he'll move the source path to the obs_*_data packages (relatively new feature in the LSST stack, currently contains defects). Until that comes together (possibly the LSST 19.0 release?), we can build them in obs_pfs and set the envvar appropriately.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a>, what are the requirements from your end? Are you OK with this living in obs_pfs? Do we really need to be able to change values based on whether they are set to a particular value, or is it enough to just set it to something new (if we can define in python code what it gets set to)?</p> <p>So we can replace all of the obs_pfs header-&gt;metadata functionaity by this? That sounds like a big win. Am a little surprised by how static the "fix_headers" mechanism is, and that it requires one file per input file, but it obviously could be extended if necessary. I am actually with you on preferring <em>code</em> for this kind of problem: a recent Subaru gotcha was that the supplied UTC timestamps were in HST for a few days; for that, being able to call out to to a function which could add 10 hours for all visits in a given range is much more sane than making a file for each.</p> <p>Realistically, DRP is the only consumer. And as far as I can tell that only cares about ~20 cards (time, identity, motors, exposure times). So sure, this looks fine.</p> <p>I've put an implementation in place that uses Tim Jenness' astro_metadata_translator package. Unfortunately it doesn't have any actual data in place for headers to patch, but I've checked that it does in fact patch the headers on ingest.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a>, please have a look, and let me know what you think. I've put your work on a new <tt>u/cloomis/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-405" title="RFC: header patching" class="issue-link" data-issue-key="PIPE2D-405"><del>PIPE2D-405</del></a></tt> branch.</p> <p>P.S. This is based on <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-442" title="Upgrade to LSST stack v18.1.0" class="issue-link" data-issue-key="PIPE2D-442"><del>PIPE2D-442</del></a> (upgrade to LSST 18.1.0), because it requires functionality that doesn't exist in LSST 16.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a> can you review please?</p> <p>Delay discussion of <tt>obs_pfs_data</tt> repo (raised during <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a> review) to a future ticket. Otherwise, review complete.</p> <p>Merged to master.</p> <p>And then discovered again that this requires LSST 18.1.0, so reverted the merge.</p> <p>Merged to master (by reverting the revert of the accidental merge).</p> Blocks PIPE2D-442 Development Rank 0|02qpt6:508yl03zc01 Reviewers cloomis Sprint 2DDRP-2019 D 2DDRP-2019 E 2DDRP-2019 F 2DDRP-2019 G Story Points 7.0