https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/INFRA-30 Software Development Infrastructure <p>Craig proposes a scheme in his email (he'll add it to this issue).</p> INFRA-30

Define filenames for 2-D inputs

Story Major Won't Fix Won't Fix rhl rhl Wed, 3 Sep 2014 14:05:47 +0000 Fri, 8 Jul 2016 14:13:35 +0000 Fri, 8 Jul 2016 14:13:35 +0000 0 4 <p>The proposed scheme for the files form the DA systems was sent out as <a href="https://pfs.ipmu.jp/pipermail/subaruif-software/2014q3/000042.html" class="external-link" rel="nofollow">https://pfs.ipmu.jp/pipermail/subaruif-software/2014q3/000042.html</a>. In summary and in approximate python syntax</p> <p>For all CCD, IR, and co-added IR image files:</p> <ul class="alternate" type="square"> <li>fileName = "%(prefix)s%(seqno)06d%(specid)1d%(camid)1d.fits"</li> <li>fullPath = os.path.join(rootDir, nightName, fileName)</li> </ul> <p>At Subaru:</p> <ul class="alternate" type="square"> <li>For CCD and co-added IR, prefix = "PFSA"</li> <li>For up-the-ramp IR data, prefix = "PFSB" (one HDU per read).</li> </ul> <p>Offsite, perhaps use PFxA and PFxB, where 'x' is different for each<br/> site. Given the distributed engineering and development, keeping<br/> sequence numbers will be hard, but with this scheme we can have<br/> globally unique and consistent names.</p> <ul class="alternate" type="square"> <li>seqno is allocated to us by Gen2. For engineering, we keep a per-site seqno.</li> <li>seqno is the same for PFSA and PFSB</li> <li>specid is (1,2,3,4). [ Engineering frames could/should use the other numbers. ]</li> <li>camid is (1,2,3) # blue, red, IR. We <em>might</em> instead separate the<br/> detectors, so (1,2, 3,4, 5) for b1,b2,r1,r2,ir<br/> ? An alternative to using PFSB for the up-the-ramp reads would be to<br/> use a distinct camid.</li> </ul> <ul class="alternate" type="square"> <li>nightName:<br/> MJD? "%Y-%m-%d"? "%Y%m%d"? <span class="error">&#91;I prefer the well-behaved integer MJD, FWIW&#93;</span><br/> Rollover hour? MJD and UTC 00:00 are ~at observatory midnight. So<br/> use JD/day+0.5?</li> </ul> <p> For example, the 123rd saved image from the spec#1 blue camera, taken<br/> yesterday at JHU:</p> <p>/data/pfs/2014-03-25/PFJA00012311.fits</p> <p> or the 1000th IR coadd and up-the-ramp frames for the 2nd spectrograph:</p> <p>/data/pfs/56741/PFSA00100023.fits<br/> /data/pfs/56741/PFSB00100023.fits</p> <p>Is this ticket still active? I suspect that there are now many example engineering exposures which already use the proposed naming scheme, and changes now are likely unnecessary and possibly confusing. So, is the proposed scheme open to debate, or should it be considered closed?</p> <p>This is the scheme we are currently using, but there is one pending change and we have no real detector data, so please suggest improvements.</p> <p>The change is to use per-site prefixes. PFS</p> {P,J,L,I} <p> for PU,JHU,LAM,IPMU. That allows independent seqnos, but does mean that we need to use a camid to indicate up-the-ramp files.</p> <p>Do we have restrictions on the style of the raw data files? Your suggestion includes an 8 digit number, and I'm wondering if that's a necessity:</p> <p>%(seqno)06d%(specid)1d%(camid)1d</p> <p>If the camid is intended to represent blue, red, or IR; would it be more intuitive when looking at filenames to use b,r,i ? I might also switch the order of specid and camid, and add a hyphens to break the 8 digit number to something easier to read:</p> <p>So the examples:</p> <p>PFJA00012311.fits<br/> PFSA00100023.fits</p> <p>would become:</p> <p>PFJA-000123-b1.fits<br/> PFSA-001000-i2.fits</p> <p>Certainly after working with the data for any time, your eye will see the coded information anyway. But if it's possible to make it immediately apparent, I think that has some value.</p> <p>I believe that the final format at Subaru must be %4s%08d.fits, with PFSA currently the only allowed prefix.</p> <p>Ahh, ok. I had suspected that might be the case. So it sounds like the only issue here is how we allocate the 8 digit number. In which case, Craig's suggestion is a fine choice.</p> <p>I there's nothing more to do here. Since the work has been done by Craig, I done feel there's any conflict of interest in me assigning it to myself to review.</p> <p>I think the relevant review points came up in the inline discussion. Done.</p> <p>This work has been moved into the data model project (DAMD), and is tracked in git at <a href="https://github.com/Subaru-PFS/datamodel/blob/master/datamodel.txt" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/datamodel/blob/master/datamodel.txt</a></p> Development Epic Link DAMD-14 Rank 0|02qpud: Reviewers bick Sprint 2014-9 2014-12 2014-13 2014-14 Story Points 1.0