https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/INFRA-32 Software Development Infrastructure <p>(e.g. ipython notebooks). N.b. google for git-filter-branch or use "BFG repo cleaner"</p> INFRA-32

Scrub drp_stella for old large files

Task Major Done Done aritter rhl Fri, 17 Jun 2016 20:41:24 +0000 Fri, 15 Jul 2016 17:09:20 +0000 Fri, 15 Jul 2016 17:09:19 +0000 0 3 <p>Hmmm... While bfg worked for the ".fits" files in the "tests" directory, it doesn't seem to work for ".fits.gz" files nor for the ".ipynb" files. Although bfg says that it deleted the files they are still there. Gonna have to try something else...</p> <p>So apparently this worked just fine:<br/> git filter-branch --force --index-filter 'git rm --cached -r --ignore-unmatch python/pfs/drp/stella/*.ipynb' --prune-empty --tag-name-filter cat &#8211; --all<br/> git filter-branch --force --index-filter 'git rm --cached -r --ignore-unmatch tests/data/calExp/2016-01-12/v000000*/PFFAr2.fits.gz' --prune-empty --tag-name-filter cat &#8211; --all<br/> git reflog expire --expire=now --all<br/> git gc --prune=now<br/> git gc --aggressive --prune=now<br/> git push origin master --force</p> <p>All fits files and not needed ipython notebooks have been removed from the repo</p> <blockquote><p>All ... and not needed ipython notebooks have been removed from the repo</p></blockquote> <p>Which notebooks are needed?</p> <p>createDetGeom.ipynb to create the detector geometry fits files</p> <p>and</p> <p>createRefSpec.ipynb to create the reference spectrum</p> <p>I don't think any executable scripts should be notebooks, please export them as straight python, and make sure that they can be run from the command line (by adding argparse argument parsing etc.). </p> <p>If you have educational notebooks, please move them into a new product.</p> <p>"createDetGeom" has been moved to a task to be executed from bin.src/createDetGeom.py (command line task didn't work because there are no data to be processed and no butler to be associated with. Error: "WARNING: Not running the task because there is no data to process; you may preview data using "--show data"")</p> <p>"createRefSpec" has been moved to a command line task</p> <p>See drp_stella tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/INFRA-32" title="Scrub drp_stella for old large files" class="issue-link" data-issue-key="INFRA-32"><del>INFRA-32</del></a></p> <p>Comments on the code at <a href="https://github.com/Subaru-PFS/drp_stella/pull/1" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/drp_stella/pull/1</a> &#8211; I think a bit more thought could make the scripts a clearer, but I don't think there are any obvious regressions here, so we should probably file another issue for refactoring.</p> <p>Confirmed that drp_stella is now pruned to a sensible size.</p> <p>Will address provenance of detector geometry in <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-37" title="Justify detector geometry constants" class="issue-link" data-issue-key="PIPE2D-37"><del>PIPE2D-37</del></a>. Other comments have been addressed.</p> Relates PIPE2D-37 Development Rank 0|02qpua: Reviewers swinbank Sprint 2014-13 2014-14