[INFRA-30] Define filenames for 2-D inputs Created: 03/Sep/14  Updated: 08/Jul/16  Resolved: 08/Jul/16

Status: Won't Fix
Project: Software Development Infrastructure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Major
Reporter: rhl Assignee: rhl
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Story Points: 1
Epic Link: Data Model
Sprint: 2014-9, 2014-12, 2014-13, 2014-14
Reviewers: bick

 Description   

Craig proposes a scheme in his email (he'll add it to this issue).



 Comments   
Comment by cloomis [ 03/Sep/14 ]

The proposed scheme for the files form the DA systems was sent out as https://pfs.ipmu.jp/pipermail/subaruif-software/2014q3/000042.html. In summary and in approximate python syntax

For all CCD, IR, and co-added IR image files:

  • fileName = "%(prefix)s%(seqno)06d%(specid)1d%(camid)1d.fits"
  • fullPath = os.path.join(rootDir, nightName, fileName)

At Subaru:

  • For CCD and co-added IR, prefix = "PFSA"
  • For up-the-ramp IR data, prefix = "PFSB" (one HDU per read).

Offsite, perhaps use PFxA and PFxB, where 'x' is different for each
site. Given the distributed engineering and development, keeping
sequence numbers will be hard, but with this scheme we can have
globally unique and consistent names.

  • seqno is allocated to us by Gen2. For engineering, we keep a per-site seqno.
  • seqno is the same for PFSA and PFSB
  • specid is (1,2,3,4). [ Engineering frames could/should use the other numbers. ]
  • camid is (1,2,3) # blue, red, IR. We might instead separate the
    detectors, so (1,2, 3,4, 5) for b1,b2,r1,r2,ir
    ? An alternative to using PFSB for the up-the-ramp reads would be to
    use a distinct camid.
  • nightName:
    MJD? "%Y-%m-%d"? "%Y%m%d"? [I prefer the well-behaved integer MJD, FWIW]
    Rollover hour? MJD and UTC 00:00 are ~at observatory midnight. So
    use JD/day+0.5?

For example, the 123rd saved image from the spec#1 blue camera, taken
yesterday at JHU:

/data/pfs/2014-03-25/PFJA00012311.fits

or the 1000th IR coadd and up-the-ramp frames for the 2nd spectrograph:

/data/pfs/56741/PFSA00100023.fits
/data/pfs/56741/PFSB00100023.fits

Comment by bick [ 10/Dec/14 ]

Is this ticket still active? I suspect that there are now many example engineering exposures which already use the proposed naming scheme, and changes now are likely unnecessary and possibly confusing. So, is the proposed scheme open to debate, or should it be considered closed?

Comment by Anonymous [ 10/Dec/14 ]

This is the scheme we are currently using, but there is one pending change and we have no real detector data, so please suggest improvements.

The change is to use per-site prefixes. PFS

{P,J,L,I}

for PU,JHU,LAM,IPMU. That allows independent seqnos, but does mean that we need to use a camid to indicate up-the-ramp files.

Comment by bick [ 11/Dec/14 ]

Do we have restrictions on the style of the raw data files? Your suggestion includes an 8 digit number, and I'm wondering if that's a necessity:

%(seqno)06d%(specid)1d%(camid)1d

If the camid is intended to represent blue, red, or IR; would it be more intuitive when looking at filenames to use b,r,i ? I might also switch the order of specid and camid, and add a hyphens to break the 8 digit number to something easier to read:

So the examples:

PFJA00012311.fits
PFSA00100023.fits

would become:

PFJA-000123-b1.fits
PFSA-001000-i2.fits

Certainly after working with the data for any time, your eye will see the coded information anyway. But if it's possible to make it immediately apparent, I think that has some value.

Comment by Anonymous [ 11/Dec/14 ]

I believe that the final format at Subaru must be %4s%08d.fits, with PFSA currently the only allowed prefix.

Comment by bick [ 11/Dec/14 ]

Ahh, ok. I had suspected that might be the case. So it sounds like the only issue here is how we allocate the 8 digit number. In which case, Craig's suggestion is a fine choice.

Comment by bick [ 11/Dec/14 ]

I there's nothing more to do here. Since the work has been done by Craig, I done feel there's any conflict of interest in me assigning it to myself to review.

Comment by bick [ 11/Dec/14 ]

I think the relevant review points came up in the inline discussion. Done.

Comment by rhl [ 08/Jul/16 ]

This work has been moved into the data model project (DAMD), and is tracked in git at https://github.com/Subaru-PFS/datamodel/blob/master/datamodel.txt

Generated at Sat Feb 10 16:48:41 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.