https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/INFRA-57 Software Development Infrastructure <p>We are going to have quite large example files within the datamodel git repo, and I expect that test data for the 1-D and 2-D pipelines may also be pretty large.</p> <p>It is very convenient to handle this sort of thing with the git code/text, but git (and in particular github) doesn't handle large binary blobs very well. Fortunately the git-lfs ("large file system") extension does do this for us; unfortunately using github's service is expensive.</p> <p>Please setup our own server to support git-lfs files within our git repos. The LSST project has done this, and can provide help as needed. If we decide to pay for github git-lfs, please provide a document discussing how we manage the data (e.g. ensuring that we don't use too much, or too much bandwidth).</p> INFRA-57

Please setup a git-lfs server

Story Major Done Done shimono rhl Sat, 3 Sep 2016 20:49:17 +0000 Sun, 25 Jun 2017 19:47:50 +0000 Sun, 25 Jun 2017 19:47:50 +0000 0 2 <p>I think that the decision on <a href="https://pfspipe.ipmu.jp/jira/browse/INFRA-31" title="Create repo for unittest data files" class="issue-link" data-issue-key="INFRA-31"><del>INFRA-31</del></a> was to buy support from github, but I'd like to see that formalised.</p> <p>We have project wide service and account information document at 'doc' repo @ github. After existing PR for upgrading to recent service modification fixed, this kind of minor fixes shall be applied.<br/> <a href="https://github.com/Subaru-PFS/doc/pull/1" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/doc/pull/1</a></p> <p>Also, we may be better to have in pfspipe web page (<a href="https://pfspipe.ipmu.jp/jira/browse/INFRA-52" title="Update web page for PFS software" class="issue-link" data-issue-key="INFRA-52"><del>INFRA-52</del></a>).</p> <p>doc/PR1 is now <a href="https://pfspipe.ipmu.jp/jira/browse/INFRA-84" title="pfs.ipmu.jp/usage: Update account request and registration part" class="issue-link" data-issue-key="INFRA-84"><del>INFRA-84</del></a>.</p> <p>If we just think about cost: </p> <p>github's plan</p> <ul> <li>$5/mo for 50GB storage and 50GB transfer</li> </ul> <p>AWS (need to have proxy for git-lfs at somewhere)</p> <ul> <li>$0.0023/GB for storage (per 128kB)</li> <li>$0.004/1000 requests</li> <li>$0.090/1GB transfer out<br/> and in $4.62/50GB for storage+transfer.</li> </ul> <p>Our usage is dominated by transfer data rate (for CI pull).<br/> If we just think about cost, AWS and github is comparable. <br/> If we can setup storage at cloud service with no transfer cost, it could be a reduce, but not large amount (as for now).</p> <p>Therefore, as TOC point of view including human resource for management, I'd propose to keep using github' git-lfs for now.</p> <p>Added a section about git-lfs in pfspipe web page.<br/> Will merge and deply to real, by 2017/06/17 if no comment supplied.</p> <p>note added, merged and deployed.</p> Relates INFRA-31 Development Rank 0|ii00n3: