PFS-JIRA https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 [SCIDB-68] prototype v2: directory structure of the 2d outputs https://pfspipe.ipmu.jp/jira/browse/SCIDB-68 Science Database <p>Come up with a tentative plan for the output directory structure.</p> SCIDB-68 prototype v2: directory structure of the 2d outputs Story Normal Done Done hassan Masayuki Tanaka Wed, 14 Nov 2018 05:02:33 +0000 Mon, 19 Nov 2018 23:53:11 +0000 Mon, 19 Nov 2018 23:53:11 +0000 0 3 <p>I discussed this with the Princeton team. In short, the directory structure is specified by you.</p> <p>PFS scripts make use of the LSST Butler concept for input/output. All input data and output products are read from/written to what is known as a 'Butler Data Repository'. On a file system, this corresponds to a single directory. For each PFS script called, this directory needs to be passed as the first argument.</p> <p>Below that directory, a number of subdirectories can be created, for calibs, and for output data. The names of those are also specified by the user when the PFS script in question is called. So you can decide the names of those too.</p> <p>See <a href="https://pipelines.lsst.io/v/DM-11034/getting-started/data-setup.html" class="external-link" rel="nofollow">https://pipelines.lsst.io/v/DM-11034/getting-started/data-setup.html</a> as an example. I understand that <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=sogo.mineo" class="user-hover" rel="sogo.mineo">sogo.mineo</a> is also aware of Butler concepts.</p> <p>Please drop me a comment or message me if I have misunderstood the request.</p> <p>The directory structure we're currently using is in specified in the <a href="https://github.com/Subaru-PFS/obs_pfs/blob/master/policy/PfsMapper.yaml" class="external-link" rel="nofollow">PfsMapper.yaml</a> file; I think this is a reasonable starting point.</p> <p>OK. I think one possible option might be to have directories like:<br/> "survey_sim/v%02d/%05d/%s/" % (simulation_version, tract,patch)<br/> and put the psfObject files there:<br/> "pfsObject-%05d-%s-%3d-%08x-%02d-0x%08x.fits" % (tract, patch, catId, objId, nVisit % 100, pfsVisitHash)<br/> Does this look reasonable?</p> <p>The <a href="https://github.com/Subaru-PFS/datamodel/blob/master/datamodel.txt#L302-L305" class="external-link" rel="nofollow">datamodel</a> currently specifies:</p> <blockquote> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-quote">"pfsObject-%05d-%s-%3d-%08x-%02d-0x%08x.fits"</span> % (tract, patch, catId, objId, nVisit % 100, pfsVisitHash) </pre> </div></div> <p>The path would be</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> tract/patch/pfsObject-*.fits </pre> </div></div></blockquote> <p>Of course, this is under a <tt>rerun</tt> directory within the data repository, so the full path would be something like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /path/to/dataRepo/rerun/&lt;rerunName&gt;/&lt;tract&gt;/&lt;patch&gt;/pfsObject-&lt;stuff&gt;.fits </pre> </div></div> <p>Please note that <b>this is subject to change</b>, for a couple of reasons:<br/> 1. There's no leading <tt>pfsObject</tt> element in the directory structure separating this data type from others, unlike for products like <a href="https://github.com/Subaru-PFS/obs_pfs/blob/master/policy/PfsMapper.yaml#L129" class="external-link" rel="nofollow">pfsArm</a>; I expect this is an oversight that we will remedy, and the reason that we haven't yet is simply because we don't yet have a component in the 2D pipeline that writes pfsObject files so the problem hasn't been noticed before now.<br/> 2. I expect the datamodel to change a bit as a result of the 2D pipeline design that is currently under internal review.</p> <p>Nevertheless, if you use the data butler to read and write the data products, then the need to know the precise directory structure vanishes, and the work required to adapt to changes is small, if any.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> /path/to/dataRepo/rerun/&lt;rerunName&gt;/&lt;tract&gt;/&lt;patch&gt;/pfsObject-&lt;stuff&gt;.fits </pre> </div></div> <p>OK, I think we should follow this. I would propose that we use a leading word 'sim_' for simulation reruns. For example, in our case:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> /path/to/dataRepo/rerun/sim_prototype_v2/&lt;tract&gt;/&lt;patch&gt;/pfsObject-&lt;stuff&gt;.fits </pre> </div></div> <p>This would allow us to distinguish between real runs from simulated ones. I understand that the data model is subject for changes (e.g., I would expect something like 'deepCoadd/' somewhere in the directory structure), but is there a way to indicate which rerun is based on which version of the data model?  If not, perhaps we should invent one?</p> <p>You can encode whatever information into the rerun name you want, but I would advise against making it complicated as the rerun name is based on policy only, so it's not guaranteed to be always be what you want.</p> <p>The LSST code writes the version information as the <tt>packages</tt> dataset. So it's not possible to tell the version that was used simply by looking at the directory structure, but it requires a little bit of python code.</p> <p>We will adopt</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> /path/to/dataRepo/rerun/sim_prototype_v2/&lt;tract&gt;/&lt;patch&gt;/pfsObject-&lt;stuff&gt;.fits</pre> </div></div> <p> </p> Development Epic Link SCIDB-64 Rank 0|s001q0: