[INSTRM-823] Restructure /data at Subaru Created: 22/Nov/19  Updated: 05/May/20  Resolved: 05/Dec/19

Status: Done
Project: Instrument control development
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Normal
Reporter: cloomis Assignee: Yoshida, Hiroshige
Resolution: Done Votes: 0
Labels: SM1, SPS, subaru-personnel
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates
relates to INSTRM-895 Document Subaru /data structure Done
relates to PIPE2D-571 Ingest into a common data repo Done
relates to INSTRM-831 Re-layer /data usage for MCS and SPS. Done
Story Points: 1
Sprint: SM1PD-2019 C

 Description   

Currently /data is a new ZFS filesystem, with the server side named /proddata because that was the name on the old server.

We probably want a slightly different structure, for clarity, efficiency, and maintainability. I suggest that we keep mounting data products under /data, but have several ZFS filesystems mounted under that. In particular:

  • /data/mcs/ (in use; old data needs to be copied over to the new server.)
  • /data/sps/ (currently named /data/pfs at LAM/JHU. Keeping that would be easier. )
  • /data/agcc/
  • /data/drp/

The raw data filesystems (mcs,sps,agcc) should have compression turned off (we use FITS compression), but any others should have the default compression on. The server filesystem names should be the same as the exported mountpoints.

All should be group pfs. /data/sps must be user pfs-data and chmod 02755; the rest should be user pfs and 02775. [ /data/

{mcs,agcc}

should also be user=pfs-data, 02755, but that needs to be a separate ticket. ].

Any other sub-directories we come up with should be their own ZFS filesystems (/data/logs, etc.)

Comments? Especially: arnaud.lefur fmadec chyan rhl philip Yoshida, Hiroshige

 Comments   
Comment by rhl [ 22/Nov/19 ]

I think I'd have transposed this, to e.g. /data/raw/2019-11-21/sps, /data/raw/2019-11-21/mcs, etc. so that complete sets of data get kept together.

Comment by cloomis [ 22/Nov/19 ]

Yes. But I suggest that having the flexibility of separate filesystems outweighs that. Maybe that is too sysadmin-y a concern.

Comment by rhl [ 22/Nov/19 ]

I'd "get it right then make it fast" – in this context do it "properly" then if we get bitten move things around and create a maze of symbolic links

Comment by cloomis [ 22/Nov/19 ]

Fair enough. So two filesystems: /data/raw/$DATE/

{mcs,sps,agcc} and /data/drp, with /data/raw being pfs-data:pfs and 02755.
Or simply pull that up and have /data/$DATE/{mcs,sps,agcc}

and /drp

Comment by Yoshida, Hiroshige [ 04/Dec/19 ]

Created and exported /data/raw and /data/drp as specified. They are already mounted on VM guests that mount /proddata on /data.

Comment by hassan [ 05/Dec/19 ]

cloomis agrees that this can be closed.

Generated at Sat Feb 10 16:29:00 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.