PFS-JIRA

PFS-JIRA https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us

8.3.4

803005

13-09-2019

[INFRA-53] Cleanup obs_pfs in git https://pfspipe.ipmu.jp/jira/browse/INFRA-53 Software Development Infrastructure <p>Please cleanup obs_pfs in git (just the way that Andreas cleaned up drp_stella)</p> <blockquote> <p>$ du -hs pfs<br/> 608M pfs<br/> $ du -hs pfs/*<br/> 4.0K pfs/README.md<br/> 4.0K pfs/SConstruct<br/> 32K pfs/bin.src<br/> 36K pfs/config<br/> 19M pfs/pfs<br/> 8.0K pfs/policy<br/> 36K pfs/python<br/> 8.0K pfs/tests<br/> 8.0K pfs/ups</p></blockquote> INFRA-53

Cleanup obs_pfs in git

Bug Major Done Done price rhl Wed, 10 Aug 2016 20:51:57 +0000 Mon, 5 Dec 2016 23:09:17 +0000 Mon, 5 Dec 2016 23:09:17 +0000 0 3 <p>This is not a mere theoretical convenience. I'm hitting space limits on Travis-CI, and trimming 500 MB would help.</p> <p>I've deleted the <tt>tickets/00*</tt> branches, plus the file <tt>redFiberTraces.fits</tt>, which was 60-something MB (in excess of the 50 MB suggested limit for GitHub repos). These changes reduce the repo size from 636 MB to 119 MB. I've put the trimmed repo up on my personal GitHub: <a href="https://github.com/PaulPrice/obs_pfs" class="external-link" rel="nofollow">https://github.com/PaulPrice/obs_pfs</a> .</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> price@price-laptop:~/temp/delete $ git clone --mirror git@github.com:Subaru-PFS/obs_pfs.git Cloning into bare repository <span class="code-quote">'obs_pfs.git'</span>... remote: Counting objects: 3889, done. remote: Compressing objects: 100% (5/5), done. remote: Total 3889 (delta 0), reused 0 (delta 0), pack-reused 3884 Receiving objects: 100% (3889/3889), 635.76 MiB | 6.76 MiB/s, done. Resolving deltas: 100% (1928/1928), done. Checking connectivity... done. price@price-laptop:~/temp/delete $ cd obs_pfs.git price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ git remote rm origin Note: Some branches outside the refs/remotes/ hierarchy were not removed; to delete them, use: git branch -d master git branch -d tickets/0008 git branch -d tickets/0014 git branch -d tickets/0020 git branch -d tickets/0024 git branch -d tickets/0026 git branch -d tickets/0028 git branch -d tickets/INFRA-36 git branch -d tickets/INFRA-42 git branch -d tickets/INSTRM-3 git branch -d tickets/PIPE2D-108 git branch -d tickets/PIPE2D-124 git branch -d tickets/PIPE2D-125 git branch -d tickets/PIPE2D-20 git branch -d tickets/PIPE2D-33 git branch -d tickets/PIPE2D-35 git branch -d tickets/PIPE2D-35-rhl git branch -d tickets/PIPE2D-37 git branch -d tickets/PIPE2D-38 git branch -d tickets/PIPE2D-43 git branch -d tickets/PIPE2D-70 git branch -d tickets/PIPE2D-77 git branch -d tickets/PIPE2D-80 git branch -d tickets/PIPE2D-90 git branch -d tickets/PIPE2D-92 git branch -d tickets/PIPE2D-98 git branch -d tickets/infra-0010 price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ git branch -D tickets/0008 tickets/0014 tickets/0020 tickets/0024 tickets/0026 tickets/0028 Deleted branch tickets/0008 (was ca70dba). Deleted branch tickets/0014 (was a4ab44d). Deleted branch tickets/0020 (was 21a573e). Deleted branch tickets/0024 (was a9674c2). Deleted branch tickets/0026 (was 6b72d5c). Deleted branch tickets/0028 (was 6311de0). price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ java -jar ~/Downloads/bfg-1.12.14.jar --delete-files redFiberTraces.fits Using repo : /Users/price/temp/delete/obs_pfs.git Found 59 objects to protect Found 1 tag-pointing refs : refs/tags/3.1 Found 31 commit-pointing refs : HEAD, refs/heads/master, refs/heads/tickets/INFRA-36, ... Protected commits ----------------- These are your <span class="code-keyword">protected</span> commits, and so their contents will NOT be altered: * commit 4ec989cf (<span class="code-keyword">protected</span> by <span class="code-quote">'HEAD'</span>) Cleaning -------- Found 307 commits Cleaning commits: 100% (307/307) Cleaning commits completed in 588 ms. Updating 1 Ref -------------- Ref Before After --------------------------------------------------- refs/heads/tickets/PIPE2D-124 | 9999241b | 715ab219 Updating references: 100% (1/1) ...Ref update completed in 40 ms. Commit Tree-Dirt History ------------------------ Earliest Latest | | ........................................................D... D = dirty commits (file tree fixed) m = modified commits (commit message or parents changed) . = clean commits (no changes to file tree) Before After ------------------------------------------- First modified commit | 9999241b | 715ab219 Last dirty commit | 9999241b | 715ab219 Deleted files ------------- Filename Git id ---------------------------------------- redFiberTraces.fits | 722d9b46 (62.8 MB) In total, 3 object ids were changed. Full details are logged here: /Users/price/temp/delete/obs_pfs.git.bfg-report/2016-11-22/16-32-26 BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive Has the BFG saved you time? Support the BFG on BountySource: https:<span class="code-comment">//j.mp/fund-bfg </span> price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ du -s . 636M . price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ git reflog expire --expire=now --all && git gc --prune=now --aggressive Counting objects: 3076, done. Delta compression using up to 8 threads. Compressing objects: 100% (2489/2489), done. Writing objects: 100% (3076/3076), done. Total 3076 (delta 1543), reused 1491 (delta 0) price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ du -s . 119M . price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ git remote add origin git@github.com:PaulPrice/obs_pfs.git price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ git push --mirror Counting objects: 3076, done. Delta compression using up to 8 threads. Compressing objects: 100% (946/946), done. Writing objects: 100% (3076/3076), 118.53 MiB | 610.00 KiB/s, done. Total 3076 (delta 1543), reused 3076 (delta 1543) remote: Resolving deltas: 100% (1543/1543), done. To git@github.com:PaulPrice/obs_pfs.git * [<span class="code-keyword">new</span> branch] master -> master * [<span class="code-keyword">new</span> branch] tickets/INFRA-36 -> tickets/INFRA-36 * [<span class="code-keyword">new</span> branch] tickets/INFRA-42 -> tickets/INFRA-42 * [<span class="code-keyword">new</span> branch] tickets/INSTRM-3 -> tickets/INSTRM-3 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-108 -> tickets/PIPE2D-108 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-124 -> tickets/PIPE2D-124 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-125 -> tickets/PIPE2D-125 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-20 -> tickets/PIPE2D-20 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-33 -> tickets/PIPE2D-33 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-35 -> tickets/PIPE2D-35 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-35-rhl -> tickets/PIPE2D-35-rhl * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-37 -> tickets/PIPE2D-37 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-38 -> tickets/PIPE2D-38 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-43 -> tickets/PIPE2D-43 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-70 -> tickets/PIPE2D-70 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-77 -> tickets/PIPE2D-77 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-80 -> tickets/PIPE2D-80 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-90 -> tickets/PIPE2D-90 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-92 -> tickets/PIPE2D-92 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-98 -> tickets/PIPE2D-98 * [<span class="code-keyword">new</span> branch] tickets/infra-0010 -> tickets/infra-0010 * [<span class="code-keyword">new</span> tag] 3.1 -> 3.1 ! [remote rejected] refs/pull/1/head -> refs/pull/1/head (deny updating a hidden ref) ! [remote rejected] refs/pull/2/head -> refs/pull/2/head (deny updating a hidden ref) ! [remote rejected] refs/pull/2/merge -> refs/pull/2/merge (deny updating a hidden ref) ! [remote rejected] refs/pull/3/head -> refs/pull/3/head (deny updating a hidden ref) ! [remote rejected] refs/pull/4/head -> refs/pull/4/head (deny updating a hidden ref) ! [remote rejected] refs/pull/5/head -> refs/pull/5/head (deny updating a hidden ref) ! [remote rejected] refs/pull/6/head -> refs/pull/6/head (deny updating a hidden ref) ! [remote rejected] refs/pull/6/merge -> refs/pull/6/merge (deny updating a hidden ref) ! [remote rejected] refs/pull/7/head -> refs/pull/7/head (deny updating a hidden ref) error: failed to push some refs to <span class="code-quote">'git@github.com:PaulPrice/obs_pfs.git'</span> </pre> </div></div> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=aritter" class="user-hover" rel="aritter">aritter</a>, would you please check that I've not deleted anything important?</p> <p>I deleted some more files (python notebook leftovers and some FITS files that are no longer used in the tests) at <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=rhl" class="user-hover" rel="rhl">rhl</a>'s urging, and it's down to 19 MB (for the repo, additional 19 MB for the checkout) now.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ java -jar ~/Downloads/bfg-1.12.14.jar --delete-files <span class="code-quote">"*.ipynb"</span> Using repo : /Users/price/temp/delete/obs_pfs.git Found 59 objects to protect Found 1 tag-pointing refs : refs/tags/3.1 Found 52 commit-pointing refs : HEAD, refs/heads/master, refs/heads/tickets/INFRA-36, ... Protected commits ----------------- These are your <span class="code-keyword">protected</span> commits, and so their contents will NOT be altered: * commit 4ec989cf (<span class="code-keyword">protected</span> by <span class="code-quote">'HEAD'</span>) Cleaning -------- Found 307 commits Cleaning commits: 100% (307/307) Cleaning commits completed in 419 ms. Updating 2 Refs --------------- Ref Before After ------------------------------------------------------------ refs/heads/tickets/infra-0010 | 3b89eb12 | 35873db1 refs/remotes/origin/tickets/infra-0010 | 3b89eb12 | 35873db1 Updating references: 100% (2/2) ...Ref update completed in 26 ms. Commit Tree-Dirt History ------------------------ Earliest Latest | | ........DDDDDDDDDDDDDDD..................................... D = dirty commits (file tree fixed) m = modified commits (commit message or parents changed) . = clean commits (no changes to file tree) Before After ------------------------------------------- First modified commit | 83b916b6 | 3c771fdb Last dirty commit | 3b89eb12 | 35873db1 Deleted files ------------- Filename Git id ----------------------------------------------------------------------------------------------------------------- GP PSF.ipynb | 0b7aa9ca (159.7 KB) Untitled0-checkpoint.ipynb | 05354e39 (10.4 MB), 69eb3e20 (11.4 MB), ... Untitled0.ipynb | a398fe88 (13.4 MB), b59676f8 (9.0 MB), ... Untitled1-checkpoint.ipynb | deef377d (26.8 KB), 52204d4b (13.6 KB), f8662246 (35.9 KB) Untitled1.ipynb | efa7b034 (40.1 KB), b30f7733 (35.1 KB), ... extractSum_vs_MkSlitFunc_vs_extractFromProfile.ipynb | d7eff341 (1.8 MB), 0fa29161 (1.8 MB), ... extract_and_extractFromProfile.ipynb | 038c8b83 (24.5 KB), edff9f8a (1.2 MB) paper.ipynb | d808667e (91.4 KB) rawspots.ipynb | e9c02bab (564.4 KB), 26227920 (562.9 KB) test_tasks.ipynb | 45e7909e (10.7 MB), e0d8c6fc (10.7 MB), ... unittests.ipynb | 0dbd739b (1.6 KB) In total, 278 object ids were changed. Full details are logged here: /Users/price/temp/delete/obs_pfs.git.bfg-report/2016-11-22/16-50-28 BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive Has the BFG saved you time? Support the BFG on BountySource: https:<span class="code-comment">//j.mp/fund-bfg </span> price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ du -s . 120M . price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ git reflog expire --expire=now --all && git gc --prune=now --aggressive Counting objects: 2974, done. Delta compression using up to 8 threads. Compressing objects: 100% (2400/2400), done. Writing objects: 100% (2974/2974), done. Total 2974 (delta 1513), reused 1324 (delta 0) price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ du -s . 61M . price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ git push --mirror --force Counting objects: 1235, done. Delta compression using up to 8 threads. Compressing objects: 100% (317/317), done. Writing objects: 100% (1235/1235), 42.51 MiB | 1.12 MiB/s, done. Total 1235 (delta 655), reused 1203 (delta 628) remote: Resolving deltas: 100% (655/655), completed with 29 local objects. To git@github.com:PaulPrice/obs_pfs.git + 3b89eb1...35873db tickets/infra-0010 -> tickets/infra-0010 (forced update) * [<span class="code-keyword">new</span> branch] origin/master -> origin/master * [<span class="code-keyword">new</span> branch] origin/tickets/INFRA-36 -> origin/tickets/INFRA-36 * [<span class="code-keyword">new</span> branch] origin/tickets/INFRA-42 -> origin/tickets/INFRA-42 * [<span class="code-keyword">new</span> branch] origin/tickets/INSTRM-3 -> origin/tickets/INSTRM-3 * [<span class="code-keyword">new</span> branch] origin/tickets/PIPE2D-108 -> origin/tickets/PIPE2D-108 * [<span class="code-keyword">new</span> branch] origin/tickets/PIPE2D-124 -> origin/tickets/PIPE2D-124 * [<span class="code-keyword">new</span> branch] origin/tickets/PIPE2D-125 -> origin/tickets/PIPE2D-125 * [<span class="code-keyword">new</span> branch] origin/tickets/PIPE2D-20 -> origin/tickets/PIPE2D-20 * [<span class="code-keyword">new</span> branch] origin/tickets/PIPE2D-33 -> origin/tickets/PIPE2D-33 * [<span class="code-keyword">new</span> branch] origin/tickets/PIPE2D-35 -> origin/tickets/PIPE2D-35 * [<span class="code-keyword">new</span> branch] origin/tickets/PIPE2D-35-rhl -> origin/tickets/PIPE2D-35-rhl * [<span class="code-keyword">new</span> branch] origin/tickets/PIPE2D-37 -> origin/tickets/PIPE2D-37 * [<span class="code-keyword">new</span> branch] origin/tickets/PIPE2D-38 -> origin/tickets/PIPE2D-38 * [<span class="code-keyword">new</span> branch] origin/tickets/PIPE2D-43 -> origin/tickets/PIPE2D-43 * [<span class="code-keyword">new</span> branch] origin/tickets/PIPE2D-70 -> origin/tickets/PIPE2D-70 * [<span class="code-keyword">new</span> branch] origin/tickets/PIPE2D-77 -> origin/tickets/PIPE2D-77 * [<span class="code-keyword">new</span> branch] origin/tickets/PIPE2D-80 -> origin/tickets/PIPE2D-80 * [<span class="code-keyword">new</span> branch] origin/tickets/PIPE2D-90 -> origin/tickets/PIPE2D-90 * [<span class="code-keyword">new</span> branch] origin/tickets/PIPE2D-92 -> origin/tickets/PIPE2D-92 * [<span class="code-keyword">new</span> branch] origin/tickets/PIPE2D-98 -> origin/tickets/PIPE2D-98 * [<span class="code-keyword">new</span> branch] origin/tickets/infra-0010 -> origin/tickets/infra-0010 ! [remote rejected] refs/pull/1/head -> refs/pull/1/head (deny updating a hidden ref) ! [remote rejected] refs/pull/2/head -> refs/pull/2/head (deny updating a hidden ref) ! [remote rejected] refs/pull/2/merge -> refs/pull/2/merge (deny updating a hidden ref) ! [remote rejected] refs/pull/3/head -> refs/pull/3/head (deny updating a hidden ref) ! [remote rejected] refs/pull/4/head -> refs/pull/4/head (deny updating a hidden ref) ! [remote rejected] refs/pull/5/head -> refs/pull/5/head (deny updating a hidden ref) ! [remote rejected] refs/pull/6/head -> refs/pull/6/head (deny updating a hidden ref) ! [remote rejected] refs/pull/6/merge -> refs/pull/6/merge (deny updating a hidden ref) ! [remote rejected] refs/pull/7/head -> refs/pull/7/head (deny updating a hidden ref) error: failed to push some refs to <span class="code-quote">'git@github.com:PaulPrice/obs_pfs.git'</span> price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ git reflog expire --expire=now --all && git gc --prune=now --aggressive price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ java -jar ~/Downloads/bfg-1.12.14.jar --delete-files <span class="code-quote">"sampled*.fits"</span> Using repo : /Users/price/temp/delete/obs_pfs.git Found 59 objects to protect Found 1 tag-pointing refs : refs/tags/3.1 Found 52 commit-pointing refs : HEAD, refs/heads/master, refs/heads/tickets/INFRA-36, ... Protected commits ----------------- These are your <span class="code-keyword">protected</span> commits, and so their contents will NOT be altered: * commit 4ec989cf (<span class="code-keyword">protected</span> by <span class="code-quote">'HEAD'</span>) Cleaning -------- Found 307 commits Cleaning commits: 100% (307/307) Cleaning commits completed in 296 ms. Updating 2 Refs --------------- Ref Before After ------------------------------------------------------------ refs/heads/tickets/infra-0010 | 35873db1 | ff32fc22 refs/remotes/origin/tickets/infra-0010 | 35873db1 | ff32fc22 Updating references: 100% (2/2) ...Ref update completed in 19 ms. Commit Tree-Dirt History ------------------------ Earliest Latest | | ............Dmmmmmmmmmm..................................... D = dirty commits (file tree fixed) m = modified commits (commit message or parents changed) . = clean commits (no changes to file tree) Before After ------------------------------------------- First modified commit | 4c08bf11 | a3ac5ce8 Last dirty commit | a2f46825 | 21719eac Deleted files ------------- Filename Git id ----------------------------------------------- sampledCombx2-IR-0-23.fits | 012ef06f (41.9 MB) sampledFlatx2-IR-0-23.fits | a0727dd1 (41.9 MB) In total, 60 object ids were changed. Full details are logged here: /Users/price/temp/delete/obs_pfs.git.bfg-report/2016-11-22/16-58-16 BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive Has the BFG saved you time? Support the BFG on BountySource: https:<span class="code-comment">//j.mp/fund-bfg </span> price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ git reflog expire --expire=now --all && git gc --prune=now --aggressive Counting objects: 2972, done. Delta compression using up to 8 threads. Compressing objects: 100% (2398/2398), done. Writing objects: 100% (2972/2972), done. Total 2972 (delta 1514), reused 1417 (delta 0) price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ du -s . 19M . price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ git push --mirror --force Counting objects: 1233, done. Delta compression using up to 8 threads. Compressing objects: 100% (314/314), done. Writing objects: 100% (1233/1233), 614.10 KiB | 0 bytes/s, done. Total 1233 (delta 656), reused 1201 (delta 629) remote: Resolving deltas: 100% (656/656), completed with 29 local objects. To git@github.com:PaulPrice/obs_pfs.git + 35873db...ff32fc2 tickets/infra-0010 -> tickets/infra-0010 (forced update) + 35873db...ff32fc2 origin/tickets/infra-0010 -> origin/tickets/infra-0010 (forced update) ! [remote rejected] refs/pull/1/head -> refs/pull/1/head (deny updating a hidden ref) ! [remote rejected] refs/pull/2/head -> refs/pull/2/head (deny updating a hidden ref) ! [remote rejected] refs/pull/2/merge -> refs/pull/2/merge (deny updating a hidden ref) ! [remote rejected] refs/pull/3/head -> refs/pull/3/head (deny updating a hidden ref) ! [remote rejected] refs/pull/4/head -> refs/pull/4/head (deny updating a hidden ref) ! [remote rejected] refs/pull/5/head -> refs/pull/5/head (deny updating a hidden ref) ! [remote rejected] refs/pull/6/head -> refs/pull/6/head (deny updating a hidden ref) ! [remote rejected] refs/pull/6/merge -> refs/pull/6/merge (deny updating a hidden ref) ! [remote rejected] refs/pull/7/head -> refs/pull/7/head (deny updating a hidden ref) error: failed to push some refs to <span class="code-quote">'git@github.com:PaulPrice/obs_pfs.git'</span> </pre> </div></div> <p>Note to self: <tt>tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-128" title="Fix config directories" class="issue-link" data-issue-key="PIPE2D-128"><del>PIPE2D-128</del></a></tt> needs to migrate as well.</p> <p>The file redFiberTraces.fits is essential for the tests to pass again (see <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-124" title="include the new redFiberTraces.fits file in the DRP" class="issue-link" data-issue-key="PIPE2D-124"><del>PIPE2D-124</del></a>), although it will become obsolete once <a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-11" title="Create or adapt file to include per-fiber+row wavelengths." class="issue-link" data-issue-key="DAMD-11"><del>DAMD-11</del></a> is implemented.</p> <p>How long would it take to complete <a href="https://pfspipe.ipmu.jp/jira/browse/DAMD-11" title="Create or adapt file to include per-fiber+row wavelengths." class="issue-link" data-issue-key="DAMD-11"><del>DAMD-11</del></a>? <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a>?</p> <p>`pfs/RedFiberPixels.fits.gz` can be deleted as it is old and will be replaced by `pfs/redFiberTraces.fits` in <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-124" title="include the new redFiberTraces.fits file in the DRP" class="issue-link" data-issue-key="PIPE2D-124"><del>PIPE2D-124</del></a></p> <p>As discussed on slack, please don't check data into drp_stella or ops_subaru. The tests should use stella_drp_data (after checking that it is available – cf. afw). And if we need to process raw data as part of the test that is fine.</p> <p>There are no data in drp_stella or obs_pfs, only the (fiber#,y)->wavelength file. There were 2 fits files in very old versions of drp_stella which can be safely removed from the github repository. I would object to processing raw data as part of the tests as the tests take a few minutes already, and I don't see any harm in keeping the processed Flat and Arc in drp_stella_data. Rebuilding these files would take a few minutes on my laptop as (as discussed on slack) we would first need to rebuild the master Bias and master Dark.<br/> I think rebuilding the Flat and Arc when cloning the repository would be okay...</p> <p>I've regenerated the cleaned repo including <tt>redFiberTraces.fits</tt> and deleting <tt>RedFiberPixels.fits.gz</tt>. <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=aritter" class="user-hover" rel="aritter">aritter</a>, would you please check the new version on my GitHub (<a href="https://github.com/PaulPrice/obs_pfs" class="external-link" rel="nofollow">https://github.com/PaulPrice/obs_pfs</a> )?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> price@price-laptop:~/temp $ git clone --mirror git@github.com:Subaru-PFS/obs_pfs.git Cloning into bare repository <span class="code-quote">'obs_pfs.git'</span>... remote: Counting objects: 3918, done. remote: Compressing objects: 100% (34/34), done. remote: Total 3918 (delta 13), reused 0 (delta 0), pack-reused 3883 Receiving objects: 100% (3918/3918), 635.76 MiB | 8.33 MiB/s, done. Resolving deltas: 100% (1941/1941), done. Checking connectivity... done. price@price-laptop:~/temp $ cd obs_pfs.git price@price-laptop:~/temp/obs_pfs.git (BARE:master) $ git remote rm origin Note: Some branches outside the refs/remotes/ hierarchy were not removed; to delete them, use: git branch -d master git branch -d tickets/0008 git branch -d tickets/0014 git branch -d tickets/0020 git branch -d tickets/0024 git branch -d tickets/0026 git branch -d tickets/0028 git branch -d tickets/INFRA-36 git branch -d tickets/INFRA-42 git branch -d tickets/INSTRM-3 git branch -d tickets/PIPE2D-108 git branch -d tickets/PIPE2D-124 git branch -d tickets/PIPE2D-125 git branch -d tickets/PIPE2D-128 git branch -d tickets/PIPE2D-20 git branch -d tickets/PIPE2D-33 git branch -d tickets/PIPE2D-35 git branch -d tickets/PIPE2D-35-rhl git branch -d tickets/PIPE2D-37 git branch -d tickets/PIPE2D-38 git branch -d tickets/PIPE2D-43 git branch -d tickets/PIPE2D-70 git branch -d tickets/PIPE2D-77 git branch -d tickets/PIPE2D-80 git branch -d tickets/PIPE2D-83 git branch -d tickets/PIPE2D-90 git branch -d tickets/PIPE2D-92 git branch -d tickets/PIPE2D-98 git branch -d tickets/infra-0010 price@price-laptop:~/temp/obs_pfs.git (BARE:master) $ git branch -D tickets/0008 tickets/0014 tickets/0020 tickets/0024 tickets/0026 tickets/0028 Deleted branch tickets/0008 (was ca70dba). Deleted branch tickets/0014 (was a4ab44d). Deleted branch tickets/0020 (was 21a573e). Deleted branch tickets/0024 (was a9674c2). Deleted branch tickets/0026 (was 6b72d5c). Deleted branch tickets/0028 (was 6311de0). price@price-laptop:~/temp/obs_pfs.git (BARE:master) $ java -jar ~/Downloads/bfg-1.12.14.jar --delete-files <span class="code-quote">"*.ipynb"</span> Using repo : /Users/price/temp/obs_pfs.git Found 58 objects to protect Found 1 tag-pointing refs : refs/tags/3.1 Found 35 commit-pointing refs : HEAD, refs/heads/master, refs/heads/tickets/INFRA-36, ... Protected commits ----------------- These are your <span class="code-keyword">protected</span> commits, and so their contents will NOT be altered: * commit 73d9dcce (<span class="code-keyword">protected</span> by <span class="code-quote">'HEAD'</span>) Cleaning -------- Found 313 commits Cleaning commits: 100% (313/313) Cleaning commits completed in 947 ms. Updating 1 Ref -------------- Ref Before After --------------------------------------------------- refs/heads/tickets/infra-0010 | 3b89eb12 | 35873db1 Updating references: 100% (1/1) ...Ref update completed in 41 ms. Commit Tree-Dirt History ------------------------ Earliest Latest | | ........DDDDDDDDDDDDDD...................................... D = dirty commits (file tree fixed) m = modified commits (commit message or parents changed) . = clean commits (no changes to file tree) Before After ------------------------------------------- First modified commit | 83b916b6 | 3c771fdb Last dirty commit | 3b89eb12 | 35873db1 Deleted files ------------- Filename Git id ----------------------------------------------------------------------------------------------------------------- GP PSF.ipynb | 0b7aa9ca (159.7 KB) Untitled0-checkpoint.ipynb | 05354e39 (10.4 MB), 69eb3e20 (11.4 MB), ... Untitled0.ipynb | a398fe88 (13.4 MB), b59676f8 (9.0 MB), ... Untitled1-checkpoint.ipynb | deef377d (26.8 KB), 52204d4b (13.6 KB), f8662246 (35.9 KB) Untitled1.ipynb | efa7b034 (40.1 KB), b30f7733 (35.1 KB), ... extractSum_vs_MkSlitFunc_vs_extractFromProfile.ipynb | d7eff341 (1.8 MB), 0fa29161 (1.8 MB), ... extract_and_extractFromProfile.ipynb | 038c8b83 (24.5 KB), edff9f8a (1.2 MB) paper.ipynb | d808667e (91.4 KB) rawspots.ipynb | e9c02bab (564.4 KB), 26227920 (562.9 KB) test_tasks.ipynb | 45e7909e (10.7 MB), e0d8c6fc (10.7 MB), ... unittests.ipynb | 0dbd739b (1.6 KB) In total, 278 object ids were changed. Full details are logged here: /Users/price/temp/obs_pfs.git.bfg-report/2016-11-28/16-02-02 BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive Has the BFG saved you time? Support the BFG on BountySource: https:<span class="code-comment">//j.mp/fund-bfg </span> price@price-laptop:~/temp/obs_pfs.git (BARE:master) $ java -jar ~/Downloads/bfg-1.12.14.jar --delete-files <span class="code-quote">"sampled*.fits"</span> Using repo : /Users/price/temp/obs_pfs.git Found 58 objects to protect Found 1 tag-pointing refs : refs/tags/3.1 Found 35 commit-pointing refs : HEAD, refs/heads/master, refs/heads/tickets/INFRA-36, ... Protected commits ----------------- These are your <span class="code-keyword">protected</span> commits, and so their contents will NOT be altered: * commit 73d9dcce (<span class="code-keyword">protected</span> by <span class="code-quote">'HEAD'</span>) Cleaning -------- Found 313 commits Cleaning commits: 100% (313/313) Cleaning commits completed in 313 ms. Updating 1 Ref -------------- Ref Before After --------------------------------------------------- refs/heads/tickets/infra-0010 | 35873db1 | ff32fc22 Updating references: 100% (1/1) ...Ref update completed in 20 ms. Commit Tree-Dirt History ------------------------ Earliest Latest | | ............Dmmmmmmmmm...................................... D = dirty commits (file tree fixed) m = modified commits (commit message or parents changed) . = clean commits (no changes to file tree) Before After ------------------------------------------- First modified commit | 4c08bf11 | a3ac5ce8 Last dirty commit | a2f46825 | 21719eac Deleted files ------------- Filename Git id ----------------------------------------------- sampledCombx2-IR-0-23.fits | 012ef06f (41.9 MB) sampledFlatx2-IR-0-23.fits | a0727dd1 (41.9 MB) In total, 60 object ids were changed. Full details are logged here: /Users/price/temp/obs_pfs.git.bfg-report/2016-11-28/16-02-17 BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive Has the BFG saved you time? Support the BFG on BountySource: https:<span class="code-comment">//j.mp/fund-bfg </span> price@price-laptop:~/temp/obs_pfs.git (BARE:master) $ java -jar ~/Downloads/bfg-1.12.14.jar --delete-files RedFiberPixels.fits.gz Using repo : /Users/price/temp/obs_pfs.git Found 58 objects to protect Found 1 tag-pointing refs : refs/tags/3.1 Found 35 commit-pointing refs : HEAD, refs/heads/master, refs/heads/tickets/INFRA-36, ... Protected commits ----------------- These are your <span class="code-keyword">protected</span> commits, and so their contents will NOT be altered: * commit 73d9dcce (<span class="code-keyword">protected</span> by <span class="code-quote">'HEAD'</span>) - contains 1 dirty file : - pfs/RedFiberPixels.fits.gz (18.1 MB) WARNING: The dirty content above may be removed from other commits, but as the *<span class="code-keyword">protected</span>* commits still use it, it will STILL exist in your repository. Details of <span class="code-keyword">protected</span> dirty content have been recorded here : /Users/price/temp/obs_pfs.git.bfg-report/2016-11-28/16-02-39/<span class="code-keyword">protected</span>-dirt/ If you *really* want <span class="code-keyword">this</span> content gone, make a manual commit that removes it, and then run the BFG on a fresh copy of your repo. Cleaning -------- Found 313 commits Cleaning commits: 100% (313/313) Cleaning commits completed in 472 ms. Updating 32 Refs ---------------- Ref Before After ------------------------------------------------------ refs/heads/master | 73d9dcce | 710594f1 refs/heads/tickets/INFRA-36 | e4011551 | b7b3c483 refs/heads/tickets/INFRA-42 | e50ef116 | 61ac51d6 refs/heads/tickets/INSTRM-3 | 2c69643a | 2bd4b6f1 refs/heads/tickets/PIPE2D-108 | 9c253e69 | d3dd9e34 refs/heads/tickets/PIPE2D-124 | 9999241b | 95ad708e refs/heads/tickets/PIPE2D-125 | 738fb2c0 | a788b4a7 refs/heads/tickets/PIPE2D-128 | 61ee8c1e | 83d89a12 refs/heads/tickets/PIPE2D-33 | f632e84e | fdf9dc86 refs/heads/tickets/PIPE2D-35 | 80214484 | 5632a3d6 refs/heads/tickets/PIPE2D-35-rhl | 696c7e52 | 58f4b53c refs/heads/tickets/PIPE2D-37 | 713ab1a5 | 63494793 refs/heads/tickets/PIPE2D-38 | 3b221530 | a3b5bd15 refs/heads/tickets/PIPE2D-43 | 2c568b72 | 3031f5c4 refs/heads/tickets/PIPE2D-70 | 2ae06cc9 | d1577bb0 ... Updating references: 100% (32/32) ...Ref update completed in 110 ms. Commit Tree-Dirt History ------------------------ Earliest Latest | | ......................DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD D = dirty commits (file tree fixed) m = modified commits (commit message or parents changed) . = clean commits (no changes to file tree) Before After ------------------------------------------- First modified commit | 830bc212 | 34103fbe Last dirty commit | e09c0de8 | 1d7dd95f Deleted files ------------- Filename Git id ------------------------------------------- RedFiberPixels.fits.gz | 2e6f5035 (18.1 MB) In total, 393 object ids were changed. Full details are logged here: /Users/price/temp/obs_pfs.git.bfg-report/2016-11-28/16-02-39 BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive Has the BFG saved you time? Support the BFG on BountySource: https:<span class="code-comment">//j.mp/fund-bfg </span> price@price-laptop:~/temp/obs_pfs.git (BARE:master) $ du -s 639M . price@price-laptop:~/temp/obs_pfs.git (BARE:master) $ git reflog expire --expire=now --all && git gc --prune=now --aggressive Counting objects: 3005, done. Delta compression using up to 8 threads. Compressing objects: 100% (2430/2430), done. Writing objects: 100% (3005/3005), done. Total 3005 (delta 1532), reused 1128 (delta 0) price@price-laptop:~/temp/obs_pfs.git (BARE:master) $ du -s 65M . price@price-laptop:~/temp/obs_pfs.git (BARE:master) $ git remote add origin git@github.com:PaulPrice/obs_pfs.git price@price-laptop:~/temp/obs_pfs.git (BARE:master) $ git push --mirror Counting objects: 3005, done. Delta compression using up to 8 threads. Compressing objects: 100% (898/898), done. Writing objects: 100% (3005/3005), 64.87 MiB | 1.38 MiB/s, done. Total 3005 (delta 1532), reused 3005 (delta 1532) remote: Resolving deltas: 100% (1532/1532), done. remote: warning: GH001: Large files detected. You may want to <span class="code-keyword">try</span> Git Large File Storage - https:<span class="code-comment">//git-lfs.github.com. </span>remote: warning: See http:<span class="code-comment">//git.io/iEPt8g <span class="code-keyword">for</span> more information. </span>remote: warning: File pfs/redFiberTraces.fits is 62.85 MB; <span class="code-keyword">this</span> is larger than GitHub's recommended maximum file size of 50.00 MB To git@github.com:PaulPrice/obs_pfs.git * [<span class="code-keyword">new</span> branch] master -> master * [<span class="code-keyword">new</span> branch] tickets/INFRA-36 -> tickets/INFRA-36 * [<span class="code-keyword">new</span> branch] tickets/INFRA-42 -> tickets/INFRA-42 * [<span class="code-keyword">new</span> branch] tickets/INSTRM-3 -> tickets/INSTRM-3 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-108 -> tickets/PIPE2D-108 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-124 -> tickets/PIPE2D-124 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-125 -> tickets/PIPE2D-125 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-128 -> tickets/PIPE2D-128 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-20 -> tickets/PIPE2D-20 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-33 -> tickets/PIPE2D-33 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-35 -> tickets/PIPE2D-35 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-35-rhl -> tickets/PIPE2D-35-rhl * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-37 -> tickets/PIPE2D-37 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-38 -> tickets/PIPE2D-38 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-43 -> tickets/PIPE2D-43 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-70 -> tickets/PIPE2D-70 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-77 -> tickets/PIPE2D-77 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-80 -> tickets/PIPE2D-80 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-83 -> tickets/PIPE2D-83 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-90 -> tickets/PIPE2D-90 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-92 -> tickets/PIPE2D-92 * [<span class="code-keyword">new</span> branch] tickets/PIPE2D-98 -> tickets/PIPE2D-98 * [<span class="code-keyword">new</span> branch] tickets/infra-0010 -> tickets/infra-0010 * [<span class="code-keyword">new</span> tag] 3.1 -> 3.1 ! [remote rejected] refs/pull/1/head -> refs/pull/1/head (deny updating a hidden ref) ! [remote rejected] refs/pull/2/head -> refs/pull/2/head (deny updating a hidden ref) ! [remote rejected] refs/pull/2/merge -> refs/pull/2/merge (deny updating a hidden ref) ! [remote rejected] refs/pull/3/head -> refs/pull/3/head (deny updating a hidden ref) ! [remote rejected] refs/pull/4/head -> refs/pull/4/head (deny updating a hidden ref) ! [remote rejected] refs/pull/5/head -> refs/pull/5/head (deny updating a hidden ref) ! [remote rejected] refs/pull/6/head -> refs/pull/6/head (deny updating a hidden ref) ! [remote rejected] refs/pull/6/merge -> refs/pull/6/merge (deny updating a hidden ref) ! [remote rejected] refs/pull/7/head -> refs/pull/7/head (deny updating a hidden ref) ! [remote rejected] refs/pull/8/head -> refs/pull/8/head (deny updating a hidden ref) ! [remote rejected] refs/pull/8/merge -> refs/pull/8/merge (deny updating a hidden ref) error: failed to push some refs to <span class="code-quote">'git@github.com:PaulPrice/obs_pfs.git'</span> </pre> </div></div> <p>Looked at the comments and couldn't find any obvious problems. Tests and runPipeline script pass. No objections to merge</p> <p>Thanks Andreas.</p> <p>Pushed to the main GitHub site and verified the reduced size. Sent a heads-up e-mail to pfs_software.</p> Development Rank 0|031vid:xii Reviewers aritter