[INSTRM-305] configure on-site storage LUNs Created: 19/Mar/18  Updated: 04/Apr/18  Resolved: 04/Apr/18

Status: Done
Project: Instrument control development
Component/s: ics_production
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Normal
Reporter: shimono Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File screenshot-1.png    
Epic Link: ICS-Storage

 Description   

Configure LUNs to on-site production storage.
Current the newest assumption seems to be in INSTRM-112 as:

  • 5TB for VM guest images (20GB each, capacity 250 for both active and backup)
  • 10TB for postgresql database storage (incl. WAL etc.)
  • 10TB for operational data incl. log, running output, etc.
  • 10TB for backup area
  • 70TB for production (FITS) data


 Comments   
Comment by shimono [ 20/Mar/18 ]

memos:

  • IR up-the ramp is 32MBx4/~5.5sec in raw, could be to ~0.1 with double incremental differential and Rice. Assuming 13MB/5.5sec continuous, rate will be 100GB per one night ~12h, 1.5TB per ~2w run.
  • online database backup (full dump or copied) is not assumed. incremental logs or some replication to external site is in a plan. having dedicate LUN for WAL is not assumed, should be in somewhere DB local (fast/flash storage) or in db storage.
  • backup area including archival outputs like rsyslog storage
Comment by shimono [ 20/Mar/18 ]

docneeded+ (to be in ics_doc?

Comment by shimono [ 20/Mar/18 ]

considering incremental backup over whole operational data LUN and keeping raw dump from various text outputs (e.g. rsyslog typed log), backup area LUN is better to be a bit larger. also we might not need such large storage for operational data as 10TB (even 200 hosts, 50GB per each), we may go as:

  • 70TB for production (FITS) data (could be parted into 5, if we want one per one IR)
  • 5TB for operational data, incl. log or running outputs from actors
  • 10TB for postgresql database storage
  • 5TB for VM guest images, including backup and archive
  • 17TB (15TB + remaining all) for backup data, incl. raw dump target of rsyslog etc.
Comment by shimono [ 20/Mar/18 ]

Comment by rhl [ 20/Mar/18 ]

What does:

IR up-the ramp is 32MBx4/~5.5sec in raw, could be to ~0.1 with double incremental differential and Rice. Assuming 13MB/5.5sec continuous

mean?  I think that the 0.1 is a compression factor, which I think is very optimistic.  These signals are noisy, and thus hard to compress.  Is this based on e.g. FMOS compression ratios (in which case I'm probably wrong!).

Comment by shimono [ 26/Mar/18 ]

compression factor ~0.1 was pointed during a meeting at Princeton from princeton colleagues, as far as I remember.
Not having done anything on existing real data.

Need to be stated at a doc in planning, very-original placed requirement (never reviewed although) was to keep all raw data within PFS summit storage for about a half year. What I wanted to point by my calculation was to check an order of data size, and not to miss inconsistency more than order.
If we take factor 1.0, 70TB data volume will be for 4 runs (4mo to 8mo), which is a bit smaller/shorter than required. If we take factor 0.4 (as ratio from usual Rice), 70TB will be 10 runs and enough over than required.

Comment by shimono [ 04/Apr/18 ]

configured and merged.

Generated at Sat Feb 10 16:23:37 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.