https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/INSTRM-305 Instrument control development <p>Configure LUNs to on-site production storage.<br/> Current the newest assumption seems to be in <a href="https://pfspipe.ipmu.jp/jira/browse/INSTRM-112" title="Develop and fix configurations of ICS CB2F storage" class="issue-link" data-issue-key="INSTRM-112"><del>INSTRM-112</del></a> as:</p> <ul> <li>5TB for VM guest images (20GB each, capacity 250 for both active and backup)</li> <li>10TB for postgresql database storage (incl. WAL etc.)</li> <li>10TB for operational data incl. log, running output, etc.</li> <li>10TB for backup area</li> <li>70TB for production (FITS) data</li> </ul> INSTRM-305

configure on-site storage LUNs

Task Normal Done Done Unassigned shimono Mon, 19 Mar 2018 11:03:33 +0000 Wed, 4 Apr 2018 02:26:51 +0000 Wed, 4 Apr 2018 02:26:51 +0000 ics_production 0 2 <p>memos:</p> <ul> <li>IR up-the ramp is 32MBx4/~5.5sec in raw, could be to ~0.1 with double incremental differential and Rice. Assuming 13MB/5.5sec continuous, rate will be 100GB per one night ~12h, 1.5TB per ~2w run.</li> <li>online database backup (full dump or copied) is not assumed. incremental logs or some replication to external site is in a plan. having dedicate LUN for WAL is not assumed, should be in somewhere DB local (fast/flash storage) or in db storage.</li> <li>backup area including archival outputs like rsyslog storage</li> </ul> <p>docneeded+ (to be in ics_doc?</p> <p>considering incremental backup over whole operational data LUN and keeping raw dump from various text outputs (e.g. rsyslog typed log), backup area LUN is better to be a bit larger. also we might not need such large storage for operational data as 10TB (even 200 hosts, 50GB per each), we may go as:</p> <ul> <li>70TB for production (FITS) data (could be parted into 5, if we want one per one IR)</li> <li>5TB for operational data, incl. log or running outputs from actors</li> <li>10TB for postgresql database storage</li> <li>5TB for VM guest images, including backup and archive</li> <li>17TB (15TB + remaining all) for backup data, incl. raw dump target of rsyslog etc.</li> </ul> <p> <span class="image-wrap" style=""><a id="11000_thumb" href="https://pfspipe.ipmu.jp/jira/secure/attachment/11000/11000_screenshot-1.png" title="screenshot-1.png" file-preview-type="image" file-preview-id="11000" file-preview-title="screenshot-1.png"><img src="https://pfspipe.ipmu.jp/jira/secure/thumbnail/11000/_thumb_11000.png" style="border: 0px solid black" /></a></span> </p> <p>What does:</p> <blockquote><p>IR up-the ramp is 32MBx4/~5.5sec in raw, could be to ~0.1 with double incremental differential and Rice. Assuming 13MB/5.5sec continuous</p></blockquote> <p>mean?  I think that the 0.1 is a compression factor, which I think is very optimistic.  These signals are noisy, and thus hard to compress.  Is this based on e.g. FMOS compression ratios (in which case I'm probably wrong!).</p> <p>compression factor ~0.1 was pointed during a meeting at Princeton from princeton colleagues, as far as I remember.<br/> Not having done anything on existing real data.</p> <p>Need to be stated at a doc in planning, very-original placed requirement (never reviewed although) was to keep all raw data within PFS summit storage for about a half year. What I wanted to point by my calculation was to check an order of data size, and not to miss inconsistency more than order.<br/> If we take factor 1.0, 70TB data volume will be for 4 runs (4mo to 8mo), which is a bit smaller/shorter than required. If we take factor 0.4 (as ratio from usual Rice), 70TB will be 10 runs and enough over than required.</p> <p>configured and merged.</p> Development Epic Link INSTRM-112 Rank 0|ii040n: