[INFRA-57] Please setup a git-lfs server Created: 04/Sep/16  Updated: 26/Jun/17  Resolved: 26/Jun/17

Status: Done
Project: Software Development Infrastructure
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Story Priority: Major
Reporter: rhl Assignee: shimono
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates
relates to INFRA-31 Create repo for unittest data files Done

 Description   

We are going to have quite large example files within the datamodel git repo, and I expect that test data for the 1-D and 2-D pipelines may also be pretty large.

It is very convenient to handle this sort of thing with the git code/text, but git (and in particular github) doesn't handle large binary blobs very well. Fortunately the git-lfs ("large file system") extension does do this for us; unfortunately using github's service is expensive.

Please setup our own server to support git-lfs files within our git repos. The LSST project has done this, and can provide help as needed. If we decide to pay for github git-lfs, please provide a document discussing how we manage the data (e.g. ensuring that we don't use too much, or too much bandwidth).

 Comments   
Comment by rhl [ 04/Sep/16 ]

I think that the decision on INFRA-31 was to buy support from github, but I'd like to see that formalised.

Comment by shimono [ 13/Sep/16 ]

We have project wide service and account information document at 'doc' repo @ github. After existing PR for upgrading to recent service modification fixed, this kind of minor fixes shall be applied.
https://github.com/Subaru-PFS/doc/pull/1

Also, we may be better to have in pfspipe web page (INFRA-52).

Comment by shimono [ 06/Dec/16 ]

doc/PR1 is now INFRA-84.

Comment by shimono [ 11/Jan/17 ]

If we just think about cost:

github's plan

  • $5/mo for 50GB storage and 50GB transfer

AWS (need to have proxy for git-lfs at somewhere)

  • $0.0023/GB for storage (per 128kB)
  • $0.004/1000 requests
  • $0.090/1GB transfer out
    and in $4.62/50GB for storage+transfer.

Our usage is dominated by transfer data rate (for CI pull).
If we just think about cost, AWS and github is comparable.
If we can setup storage at cloud service with no transfer cost, it could be a reduce, but not large amount (as for now).

Therefore, as TOC point of view including human resource for management, I'd propose to keep using github' git-lfs for now.

Comment by shimono [ 14/Jun/17 ]

Added a section about git-lfs in pfspipe web page.
Will merge and deply to real, by 2017/06/17 if no comment supplied.

Comment by shimono [ 26/Jun/17 ]

note added, merged and deployed.

Generated at Sat Feb 10 16:48:58 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.