https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/INFRA-31 Software Development Infrastructure <p>We need an optionally installed repository for inconveniently big data files (FITS, etc) which some unittests need. Like 'afwdata' in HSC.</p> <p>This ticket is intended to elicit HSC/LSST experience. In particular, is one repo enough (pfs_testdata?) Or will be want more (drp_testdata, ics_testdata for the camera actors, etc.)? I'd vote for one.</p> INFRA-31

Create repo for unittest data files

Task Major Done Done aritter cloomis Fri, 20 Feb 2015 19:48:32 +0000 Sat, 3 Sep 2016 20:53:17 +0000 Wed, 6 Jul 2016 06:24:06 +0000 0 6 <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a>, <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=aritter" class="user-hover" rel="aritter">aritter</a> &#8211; looks like Andreas marked this in progress, but JIRA reckons it's assigned to Craig. Should we reassign it?</p> <p>LSST, for what it's worth, has adopted git-lfs for medium-sized test data repositories (repositories with tiny data files still tend to end up in plain old git, and it's clear that git-lfs doesn't scale to really big data volumes).</p> <p>We've seen a number of issues with the complexity of git-lfs setup and usage. Just the other day, for example: <a href="https://community.lsst.org/t/issues-using-git-lfs-for-cloning-test-data-repositories/880/14" class="external-link" rel="nofollow">https://community.lsst.org/t/issues-using-git-lfs-for-cloning-test-data-repositories/880/14</a></p> <p>However, that is largely because LSST uses the git-lfs protocol, but based on its own infrastructure. Using git-lfs on GitHub's infrastructure should be much more seamless. It's free, for modest data volumes (1 GB total storage, 1 GB/month bandwidth). Would that be usable for PFS?</p> <p>Actually I've migrated from local gitolite to github using github git-lfs, and PFS has its storage. From infra point of view, we can. Policy need to be defined. </p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=aritter" class="user-hover" rel="aritter">aritter</a> tells me that, <a href="https://pfs.ipmu.jp/pipermail/allhands-software/2016-June/000451.html" class="external-link" rel="nofollow">per discussion on allhands-software</a> he intends to use a plain git repository in this case.</p> <p>In general, though, being able to use git-lfs seems like an excellent thing. How would we go about defining a policy?</p> <p>There's a little problem using gzipped files - the butler can't find them (see <a href="https://jira.lsstcorp.org/browse/DM-4924" class="external-link" rel="nofollow">https://jira.lsstcorp.org/browse/DM-4924</a>). Given the file size (~160MB) and standard-git's file size limit of 100MB the fits files need to be compressed. Should we mark <a href="https://jira.lsstcorp.org/browse/DM-4924" class="external-link" rel="nofollow">https://jira.lsstcorp.org/browse/DM-4924</a> as blocking <a href="https://pfspipe.ipmu.jp/jira/browse/INFRA-31" title="Create repo for unittest data files" class="issue-link" data-issue-key="INFRA-31"><del>INFRA-31</del></a>, or should we go to git-lfs instead?</p> <p>For the discussion of <a href="https://jira.lsstcorp.org/browse/DM-4924" class="external-link" rel="nofollow">https://jira.lsstcorp.org/browse/DM-4924</a> see <a href="https://community.lsst.org/t/how-does-the-butler-support-compression/502/15" class="external-link" rel="nofollow">https://community.lsst.org/t/how-does-the-butler-support-compression/502/15</a></p> <p>The Butler work has't been scheduled on LSST: it'll have to happen sometime, but we can't say when. Plus (per RHL in the e-mail thread linked above) git-lfs is the "correct solution" and (if we're using GitHub infrastructure) I think it should be relatively painless.</p> <p>Given all that, I suggest going ahead with git-lfs.</p> <p>Created repo "drp_stella_data" using git lfs.<br/> For changes in "drp_stella" see tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/INFRA-31" title="Create repo for unittest data files" class="issue-link" data-issue-key="INFRA-31"><del>INFRA-31</del></a>.<br/> Tests are only executed if "drp_stella_data" is setup.<br/> Edited test programs for the new file location.</p> <p>Confirm that the drp_stella_data repository has data stored using git-lfs. However, you've not included a <tt>.gitattributes</tt> file telling git-lfs to manage the data, so the won't get fetched when you clone the repository. I've created the branch <a href="https://github.com/Subaru-PFS/drp_stella_data/tree/u/swinbank/lfs" class="external-link" rel="nofollow"><tt>u/swinbank/lfs</tt></a> which contains an appropriate <tt>.gitattributes</tt>. Could you please check it and, if you're happy, merge it to <tt>master</tt>?</p> <p>I'm not sure I've understood <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=atsushi.shimono" class="user-hover" rel="atsushi.shimono">shimono</a>'s comment about defining policy for this. Did you discuss with him whether this was appropriate? Do we know how much quota we have available for this data?</p> <p>It looks like your changes to drp_stella have already been merged to <tt>master</tt>. That's fine, since I don't think we've discussed or formalized a policy for code review. In future, we should hold them on the ticket branch until the reviewer has had chance to have a look, and then, when both you and the reviewer are happy, you can merge.</p> <p>I did go through and add some comments to your commit here: <a href="https://github.com/Subaru-PFS/drp_stella/commit/aa634c375f77287bf38783db93dc36536b2ecefb#diff-cd53403f864c50438a9c2afd10f43148R34" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/drp_stella/commit/aa634c375f77287bf38783db93dc36536b2ecefb#diff-cd53403f864c50438a9c2afd10f43148R34</a>. Most of them are just minor ways in which the code could be cleaner, but there are a couple of workflow issues we should talk about.</p> <p>All comments applied and merged with master</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=atsushi.shimono" class="user-hover" rel="atsushi.shimono">shimono</a> We think that you had some thoughts about git-lfs. Do you have a problem with this solution?</p> <p>Sorry for slow response.<br/> For quota, it's only $5/mo/50GB (bandwidth or data stored), so I think we could keep in github on this moment with paying multiple (e.g. 10 of 50GB; mainly for bandwidth?). <br/> On production (e.g. during running SSP), we might have issue on bandwidth (download from many collaborators), but we could do with offline (from git) download site, and need to be discussed on such policy once we have such issue.</p> <p>In case it's helpful, here's an LSST document on what they did: <a href="https://sqr-001.lsst.io" class="external-link" rel="nofollow">https://sqr-001.lsst.io</a></p> Blocks PIPE2D-26 Relates INFRA-57 Development Rank 0|02qpu6: Reviewers swinbank Sprint 2014-13 2014-14