[#INFRA-53] Cleanup obs

Comment by price [ 22/Nov/16 ]

This is not a mere theoretical convenience. I'm hitting space limits on Travis-CI, and trimming 500 MB would help.

Comment by price [ 23/Nov/16 ]

I've deleted the tickets/00* branches, plus the file redFiberTraces.fits, which was 60-something MB (in excess of the 50 MB suggested limit for GitHub repos). These changes reduce the repo size from 636 MB to 119 MB. I've put the trimmed repo up on my personal GitHub: https://github.com/PaulPrice/obs_pfs .

price@price-laptop:~/temp/delete $ git clone --mirror git@github.com:Subaru-PFS/obs_pfs.git
Cloning into bare repository 'obs_pfs.git'...
remote: Counting objects: 3889, done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 3889 (delta 0), reused 0 (delta 0), pack-reused 3884
Receiving objects: 100% (3889/3889), 635.76 MiB | 6.76 MiB/s, done.
Resolving deltas: 100% (1928/1928), done.
Checking connectivity... done.
price@price-laptop:~/temp/delete $ cd obs_pfs.git
price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ git remote rm origin
Note: Some branches outside the refs/remotes/ hierarchy were not removed;
to delete them, use:
  git branch -d master
  git branch -d tickets/0008
  git branch -d tickets/0014
  git branch -d tickets/0020
  git branch -d tickets/0024
  git branch -d tickets/0026
  git branch -d tickets/0028
  git branch -d tickets/INFRA-36
  git branch -d tickets/INFRA-42
  git branch -d tickets/INSTRM-3
  git branch -d tickets/PIPE2D-108
  git branch -d tickets/PIPE2D-124
  git branch -d tickets/PIPE2D-125
  git branch -d tickets/PIPE2D-20
  git branch -d tickets/PIPE2D-33
  git branch -d tickets/PIPE2D-35
  git branch -d tickets/PIPE2D-35-rhl
  git branch -d tickets/PIPE2D-37
  git branch -d tickets/PIPE2D-38
  git branch -d tickets/PIPE2D-43
  git branch -d tickets/PIPE2D-70
  git branch -d tickets/PIPE2D-77
  git branch -d tickets/PIPE2D-80
  git branch -d tickets/PIPE2D-90
  git branch -d tickets/PIPE2D-92
  git branch -d tickets/PIPE2D-98
  git branch -d tickets/infra-0010
price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ git branch -D tickets/0008 tickets/0014 tickets/0020 tickets/0024 tickets/0026 tickets/0028
Deleted branch tickets/0008 (was ca70dba).
Deleted branch tickets/0014 (was a4ab44d).
Deleted branch tickets/0020 (was 21a573e).
Deleted branch tickets/0024 (was a9674c2).
Deleted branch tickets/0026 (was 6b72d5c).
Deleted branch tickets/0028 (was 6311de0).
price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ java -jar ~/Downloads/bfg-1.12.14.jar --delete-files redFiberTraces.fits

Using repo : /Users/price/temp/delete/obs_pfs.git

Found 59 objects to protect
Found 1 tag-pointing refs : refs/tags/3.1
Found 31 commit-pointing refs : HEAD, refs/heads/master, refs/heads/tickets/INFRA-36, ...

Protected commits
-----------------

These are your protected commits, and so their contents will NOT be altered:

 * commit 4ec989cf (protected by 'HEAD')

Cleaning
--------

Found 307 commits
Cleaning commits:       100% (307/307)
Cleaning commits completed in 588 ms.

Updating 1 Ref
--------------

	Ref                             Before     After   
	---------------------------------------------------
	refs/heads/tickets/PIPE2D-124 | 9999241b | 715ab219

Updating references:    100% (1/1)
...Ref update completed in 40 ms.

Commit Tree-Dirt History
------------------------

	Earliest                                              Latest
	|                                                          |
	........................................................D...

	D = dirty commits (file tree fixed)
	m = modified commits (commit message or parents changed)
	. = clean commits (no changes to file tree)

	                        Before     After   
	-------------------------------------------
	First modified commit | 9999241b | 715ab219
	Last dirty commit     | 9999241b | 715ab219

Deleted files
-------------

	Filename              Git id            
	----------------------------------------
	redFiberTraces.fits | 722d9b46 (62.8 MB)


In total, 3 object ids were changed. Full details are logged here:

	/Users/price/temp/delete/obs_pfs.git.bfg-report/2016-11-22/16-32-26

BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive


Has the BFG saved you time?  Support the BFG on BountySource:  https://j.mp/fund-bfg


price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ du -s .
636M	.
price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ git reflog expire --expire=now --all && git gc --prune=now --aggressive
Counting objects: 3076, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2489/2489), done.
Writing objects: 100% (3076/3076), done.
Total 3076 (delta 1543), reused 1491 (delta 0)
price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ du -s .
119M	.
price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ git remote add origin git@github.com:PaulPrice/obs_pfs.git
price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ git push --mirror
Counting objects: 3076, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (946/946), done.
Writing objects: 100% (3076/3076), 118.53 MiB | 610.00 KiB/s, done.
Total 3076 (delta 1543), reused 3076 (delta 1543)
remote: Resolving deltas: 100% (1543/1543), done.
To git@github.com:PaulPrice/obs_pfs.git
 * [new branch]      master -> master
 * [new branch]      tickets/INFRA-36 -> tickets/INFRA-36
 * [new branch]      tickets/INFRA-42 -> tickets/INFRA-42
 * [new branch]      tickets/INSTRM-3 -> tickets/INSTRM-3
 * [new branch]      tickets/PIPE2D-108 -> tickets/PIPE2D-108
 * [new branch]      tickets/PIPE2D-124 -> tickets/PIPE2D-124
 * [new branch]      tickets/PIPE2D-125 -> tickets/PIPE2D-125
 * [new branch]      tickets/PIPE2D-20 -> tickets/PIPE2D-20
 * [new branch]      tickets/PIPE2D-33 -> tickets/PIPE2D-33
 * [new branch]      tickets/PIPE2D-35 -> tickets/PIPE2D-35
 * [new branch]      tickets/PIPE2D-35-rhl -> tickets/PIPE2D-35-rhl
 * [new branch]      tickets/PIPE2D-37 -> tickets/PIPE2D-37
 * [new branch]      tickets/PIPE2D-38 -> tickets/PIPE2D-38
 * [new branch]      tickets/PIPE2D-43 -> tickets/PIPE2D-43
 * [new branch]      tickets/PIPE2D-70 -> tickets/PIPE2D-70
 * [new branch]      tickets/PIPE2D-77 -> tickets/PIPE2D-77
 * [new branch]      tickets/PIPE2D-80 -> tickets/PIPE2D-80
 * [new branch]      tickets/PIPE2D-90 -> tickets/PIPE2D-90
 * [new branch]      tickets/PIPE2D-92 -> tickets/PIPE2D-92
 * [new branch]      tickets/PIPE2D-98 -> tickets/PIPE2D-98
 * [new branch]      tickets/infra-0010 -> tickets/infra-0010
 * [new tag]         3.1 -> 3.1
 ! [remote rejected] refs/pull/1/head -> refs/pull/1/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/2/head -> refs/pull/2/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/2/merge -> refs/pull/2/merge (deny updating a hidden ref)
 ! [remote rejected] refs/pull/3/head -> refs/pull/3/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/4/head -> refs/pull/4/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/5/head -> refs/pull/5/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/6/head -> refs/pull/6/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/6/merge -> refs/pull/6/merge (deny updating a hidden ref)
 ! [remote rejected] refs/pull/7/head -> refs/pull/7/head (deny updating a hidden ref)
error: failed to push some refs to 'git@github.com:PaulPrice/obs_pfs.git'

Comment by price [ 23/Nov/16 ]

aritter, would you please check that I've not deleted anything important?

Comment by price [ 23/Nov/16 ]

I deleted some more files (python notebook leftovers and some FITS files that are no longer used in the tests) at rhl's urging, and it's down to 19 MB (for the repo, additional 19 MB for the checkout) now.

price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ java -jar ~/Downloads/bfg-1.12.14.jar --delete-files "*.ipynb"

Using repo : /Users/price/temp/delete/obs_pfs.git

Found 59 objects to protect
Found 1 tag-pointing refs : refs/tags/3.1
Found 52 commit-pointing refs : HEAD, refs/heads/master, refs/heads/tickets/INFRA-36, ...

Protected commits
-----------------

These are your protected commits, and so their contents will NOT be altered:

 * commit 4ec989cf (protected by 'HEAD')

Cleaning
--------

Found 307 commits
Cleaning commits:       100% (307/307)
Cleaning commits completed in 419 ms.

Updating 2 Refs
---------------

	Ref                                      Before     After   
	------------------------------------------------------------
	refs/heads/tickets/infra-0010          | 3b89eb12 | 35873db1
	refs/remotes/origin/tickets/infra-0010 | 3b89eb12 | 35873db1

Updating references:    100% (2/2)
...Ref update completed in 26 ms.

Commit Tree-Dirt History
------------------------

	Earliest                                              Latest
	|                                                          |
	........DDDDDDDDDDDDDDD.....................................

	D = dirty commits (file tree fixed)
	m = modified commits (commit message or parents changed)
	. = clean commits (no changes to file tree)

	                        Before     After   
	-------------------------------------------
	First modified commit | 83b916b6 | 3c771fdb
	Last dirty commit     | 3b89eb12 | 35873db1

Deleted files
-------------

	Filename                                               Git id                                                    
	-----------------------------------------------------------------------------------------------------------------
	GP PSF.ipynb                                         | 0b7aa9ca (159.7 KB)                                       
	Untitled0-checkpoint.ipynb                           | 05354e39 (10.4 MB), 69eb3e20 (11.4 MB), ...               
	Untitled0.ipynb                                      | a398fe88 (13.4 MB), b59676f8 (9.0 MB), ...                
	Untitled1-checkpoint.ipynb                           | deef377d (26.8 KB), 52204d4b (13.6 KB), f8662246 (35.9 KB)
	Untitled1.ipynb                                      | efa7b034 (40.1 KB), b30f7733 (35.1 KB), ...               
	extractSum_vs_MkSlitFunc_vs_extractFromProfile.ipynb | d7eff341 (1.8 MB), 0fa29161 (1.8 MB), ...                 
	extract_and_extractFromProfile.ipynb                 | 038c8b83 (24.5 KB), edff9f8a (1.2 MB)                     
	paper.ipynb                                          | d808667e (91.4 KB)                                        
	rawspots.ipynb                                       | e9c02bab (564.4 KB), 26227920 (562.9 KB)                  
	test_tasks.ipynb                                     | 45e7909e (10.7 MB), e0d8c6fc (10.7 MB), ...               
	unittests.ipynb                                      | 0dbd739b (1.6 KB)                                         


In total, 278 object ids were changed. Full details are logged here:

	/Users/price/temp/delete/obs_pfs.git.bfg-report/2016-11-22/16-50-28

BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive


Has the BFG saved you time?  Support the BFG on BountySource:  https://j.mp/fund-bfg


price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ du -s .
120M	.
price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ git reflog expire --expire=now --all && git gc --prune=now --aggressive
Counting objects: 2974, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2400/2400), done.
Writing objects: 100% (2974/2974), done.
Total 2974 (delta 1513), reused 1324 (delta 0)
price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ du -s .
 61M	.
price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ git push --mirror --force
Counting objects: 1235, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (317/317), done.
Writing objects: 100% (1235/1235), 42.51 MiB | 1.12 MiB/s, done.
Total 1235 (delta 655), reused 1203 (delta 628)
remote: Resolving deltas: 100% (655/655), completed with 29 local objects.
To git@github.com:PaulPrice/obs_pfs.git
 + 3b89eb1...35873db tickets/infra-0010 -> tickets/infra-0010 (forced update)
 * [new branch]      origin/master -> origin/master
 * [new branch]      origin/tickets/INFRA-36 -> origin/tickets/INFRA-36
 * [new branch]      origin/tickets/INFRA-42 -> origin/tickets/INFRA-42
 * [new branch]      origin/tickets/INSTRM-3 -> origin/tickets/INSTRM-3
 * [new branch]      origin/tickets/PIPE2D-108 -> origin/tickets/PIPE2D-108
 * [new branch]      origin/tickets/PIPE2D-124 -> origin/tickets/PIPE2D-124
 * [new branch]      origin/tickets/PIPE2D-125 -> origin/tickets/PIPE2D-125
 * [new branch]      origin/tickets/PIPE2D-20 -> origin/tickets/PIPE2D-20
 * [new branch]      origin/tickets/PIPE2D-33 -> origin/tickets/PIPE2D-33
 * [new branch]      origin/tickets/PIPE2D-35 -> origin/tickets/PIPE2D-35
 * [new branch]      origin/tickets/PIPE2D-35-rhl -> origin/tickets/PIPE2D-35-rhl
 * [new branch]      origin/tickets/PIPE2D-37 -> origin/tickets/PIPE2D-37
 * [new branch]      origin/tickets/PIPE2D-38 -> origin/tickets/PIPE2D-38
 * [new branch]      origin/tickets/PIPE2D-43 -> origin/tickets/PIPE2D-43
 * [new branch]      origin/tickets/PIPE2D-70 -> origin/tickets/PIPE2D-70
 * [new branch]      origin/tickets/PIPE2D-77 -> origin/tickets/PIPE2D-77
 * [new branch]      origin/tickets/PIPE2D-80 -> origin/tickets/PIPE2D-80
 * [new branch]      origin/tickets/PIPE2D-90 -> origin/tickets/PIPE2D-90
 * [new branch]      origin/tickets/PIPE2D-92 -> origin/tickets/PIPE2D-92
 * [new branch]      origin/tickets/PIPE2D-98 -> origin/tickets/PIPE2D-98
 * [new branch]      origin/tickets/infra-0010 -> origin/tickets/infra-0010
 ! [remote rejected] refs/pull/1/head -> refs/pull/1/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/2/head -> refs/pull/2/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/2/merge -> refs/pull/2/merge (deny updating a hidden ref)
 ! [remote rejected] refs/pull/3/head -> refs/pull/3/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/4/head -> refs/pull/4/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/5/head -> refs/pull/5/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/6/head -> refs/pull/6/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/6/merge -> refs/pull/6/merge (deny updating a hidden ref)
 ! [remote rejected] refs/pull/7/head -> refs/pull/7/head (deny updating a hidden ref)
error: failed to push some refs to 'git@github.com:PaulPrice/obs_pfs.git'
price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ git reflog expire --expire=now --all && git gc --prune=now --aggressive
price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ java -jar ~/Downloads/bfg-1.12.14.jar --delete-files "sampled*.fits"

Using repo : /Users/price/temp/delete/obs_pfs.git

Found 59 objects to protect
Found 1 tag-pointing refs : refs/tags/3.1
Found 52 commit-pointing refs : HEAD, refs/heads/master, refs/heads/tickets/INFRA-36, ...

Protected commits
-----------------

These are your protected commits, and so their contents will NOT be altered:

 * commit 4ec989cf (protected by 'HEAD')

Cleaning
--------

Found 307 commits
Cleaning commits:       100% (307/307)
Cleaning commits completed in 296 ms.

Updating 2 Refs
---------------

	Ref                                      Before     After   
	------------------------------------------------------------
	refs/heads/tickets/infra-0010          | 35873db1 | ff32fc22
	refs/remotes/origin/tickets/infra-0010 | 35873db1 | ff32fc22

Updating references:    100% (2/2)
...Ref update completed in 19 ms.

Commit Tree-Dirt History
------------------------

	Earliest                                              Latest
	|                                                          |
	............Dmmmmmmmmmm.....................................

	D = dirty commits (file tree fixed)
	m = modified commits (commit message or parents changed)
	. = clean commits (no changes to file tree)

	                        Before     After   
	-------------------------------------------
	First modified commit | 4c08bf11 | a3ac5ce8
	Last dirty commit     | a2f46825 | 21719eac

Deleted files
-------------

	Filename                     Git id            
	-----------------------------------------------
	sampledCombx2-IR-0-23.fits | 012ef06f (41.9 MB)
	sampledFlatx2-IR-0-23.fits | a0727dd1 (41.9 MB)


In total, 60 object ids were changed. Full details are logged here:

	/Users/price/temp/delete/obs_pfs.git.bfg-report/2016-11-22/16-58-16

BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive


Has the BFG saved you time?  Support the BFG on BountySource:  https://j.mp/fund-bfg


price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ git reflog expire --expire=now --all && git gc --prune=now --aggressive
Counting objects: 2972, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2398/2398), done.
Writing objects: 100% (2972/2972), done.
Total 2972 (delta 1514), reused 1417 (delta 0)
price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ du -s .
 19M	.
price@price-laptop:~/temp/delete/obs_pfs.git (BARE:master) $ git push --mirror --force
Counting objects: 1233, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (314/314), done.
Writing objects: 100% (1233/1233), 614.10 KiB | 0 bytes/s, done.
Total 1233 (delta 656), reused 1201 (delta 629)
remote: Resolving deltas: 100% (656/656), completed with 29 local objects.
To git@github.com:PaulPrice/obs_pfs.git
 + 35873db...ff32fc2 tickets/infra-0010 -> tickets/infra-0010 (forced update)
 + 35873db...ff32fc2 origin/tickets/infra-0010 -> origin/tickets/infra-0010 (forced update)
 ! [remote rejected] refs/pull/1/head -> refs/pull/1/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/2/head -> refs/pull/2/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/2/merge -> refs/pull/2/merge (deny updating a hidden ref)
 ! [remote rejected] refs/pull/3/head -> refs/pull/3/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/4/head -> refs/pull/4/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/5/head -> refs/pull/5/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/6/head -> refs/pull/6/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/6/merge -> refs/pull/6/merge (deny updating a hidden ref)
 ! [remote rejected] refs/pull/7/head -> refs/pull/7/head (deny updating a hidden ref)
error: failed to push some refs to 'git@github.com:PaulPrice/obs_pfs.git'

Comment by price [ 23/Nov/16 ]

Note to self: tickets/PIPE2D-128 needs to migrate as well.

Comment by aritter [ 23/Nov/16 ]

The file redFiberTraces.fits is essential for the tests to pass again (see ~~PIPE2D-124~~), although it will become obsolete once ~~DAMD-11~~ is implemented.

Comment by price [ 23/Nov/16 ]

How long would it take to complete ~~DAMD-11~~? cloomis?

Comment by aritter [ 23/Nov/16 ]

`pfs/RedFiberPixels.fits.gz` can be deleted as it is old and will be replaced by `pfs/redFiberTraces.fits` in ~~PIPE2D-124~~

Comment by rhl [ 23/Nov/16 ]

As discussed on slack, please don't check data into drp_stella or ops_subaru. The tests should use stella_drp_data (after checking that it is available – cf. afw). And if we need to process raw data as part of the test that is fine.

Comment by aritter [ 23/Nov/16 ]

There are no data in drp_stella or obs_pfs, only the (fiber#,y)->wavelength file. There were 2 fits files in very old versions of drp_stella which can be safely removed from the github repository. I would object to processing raw data as part of the tests as the tests take a few minutes already, and I don't see any harm in keeping the processed Flat and Arc in drp_stella_data. Rebuilding these files would take a few minutes on my laptop as (as discussed on slack) we would first need to rebuild the master Bias and master Dark.
I think rebuilding the Flat and Arc when cloning the repository would be okay...

Comment by price [ 29/Nov/16 ]

I've regenerated the cleaned repo including redFiberTraces.fits and deleting RedFiberPixels.fits.gz. aritter, would you please check the new version on my GitHub (https://github.com/PaulPrice/obs_pfs )?

price@price-laptop:~/temp $ git clone --mirror git@github.com:Subaru-PFS/obs_pfs.git
Cloning into bare repository 'obs_pfs.git'...
remote: Counting objects: 3918, done.
remote: Compressing objects: 100% (34/34), done.
remote: Total 3918 (delta 13), reused 0 (delta 0), pack-reused 3883
Receiving objects: 100% (3918/3918), 635.76 MiB | 8.33 MiB/s, done.
Resolving deltas: 100% (1941/1941), done.
Checking connectivity... done.
price@price-laptop:~/temp $ cd obs_pfs.git
price@price-laptop:~/temp/obs_pfs.git (BARE:master) $ git remote rm origin
Note: Some branches outside the refs/remotes/ hierarchy were not removed;
to delete them, use:
  git branch -d master
  git branch -d tickets/0008
  git branch -d tickets/0014
  git branch -d tickets/0020
  git branch -d tickets/0024
  git branch -d tickets/0026
  git branch -d tickets/0028
  git branch -d tickets/INFRA-36
  git branch -d tickets/INFRA-42
  git branch -d tickets/INSTRM-3
  git branch -d tickets/PIPE2D-108
  git branch -d tickets/PIPE2D-124
  git branch -d tickets/PIPE2D-125
  git branch -d tickets/PIPE2D-128
  git branch -d tickets/PIPE2D-20
  git branch -d tickets/PIPE2D-33
  git branch -d tickets/PIPE2D-35
  git branch -d tickets/PIPE2D-35-rhl
  git branch -d tickets/PIPE2D-37
  git branch -d tickets/PIPE2D-38
  git branch -d tickets/PIPE2D-43
  git branch -d tickets/PIPE2D-70
  git branch -d tickets/PIPE2D-77
  git branch -d tickets/PIPE2D-80
  git branch -d tickets/PIPE2D-83
  git branch -d tickets/PIPE2D-90
  git branch -d tickets/PIPE2D-92
  git branch -d tickets/PIPE2D-98
  git branch -d tickets/infra-0010
price@price-laptop:~/temp/obs_pfs.git (BARE:master) $ git branch -D tickets/0008 tickets/0014 tickets/0020 tickets/0024 tickets/0026 tickets/0028
Deleted branch tickets/0008 (was ca70dba).
Deleted branch tickets/0014 (was a4ab44d).
Deleted branch tickets/0020 (was 21a573e).
Deleted branch tickets/0024 (was a9674c2).
Deleted branch tickets/0026 (was 6b72d5c).
Deleted branch tickets/0028 (was 6311de0).
price@price-laptop:~/temp/obs_pfs.git (BARE:master) $ java -jar ~/Downloads/bfg-1.12.14.jar --delete-files "*.ipynb"

Using repo : /Users/price/temp/obs_pfs.git

Found 58 objects to protect
Found 1 tag-pointing refs : refs/tags/3.1
Found 35 commit-pointing refs : HEAD, refs/heads/master, refs/heads/tickets/INFRA-36, ...

Protected commits
-----------------

These are your protected commits, and so their contents will NOT be altered:

 * commit 73d9dcce (protected by 'HEAD')

Cleaning
--------

Found 313 commits
Cleaning commits:       100% (313/313)
Cleaning commits completed in 947 ms.

Updating 1 Ref
--------------

  Ref                             Before     After   
  ---------------------------------------------------
  refs/heads/tickets/infra-0010 | 3b89eb12 | 35873db1

Updating references:    100% (1/1)
...Ref update completed in 41 ms.

Commit Tree-Dirt History
------------------------

  Earliest                                              Latest
  |                                                          |
  ........DDDDDDDDDDDDDD......................................

  D = dirty commits (file tree fixed)
  m = modified commits (commit message or parents changed)
  . = clean commits (no changes to file tree)

                          Before     After   
  -------------------------------------------
  First modified commit | 83b916b6 | 3c771fdb
  Last dirty commit     | 3b89eb12 | 35873db1

Deleted files
-------------

  Filename                                               Git id                                                    
  -----------------------------------------------------------------------------------------------------------------
  GP PSF.ipynb                                         | 0b7aa9ca (159.7 KB)                                       
  Untitled0-checkpoint.ipynb                           | 05354e39 (10.4 MB), 69eb3e20 (11.4 MB), ...               
  Untitled0.ipynb                                      | a398fe88 (13.4 MB), b59676f8 (9.0 MB), ...                
  Untitled1-checkpoint.ipynb                           | deef377d (26.8 KB), 52204d4b (13.6 KB), f8662246 (35.9 KB)
  Untitled1.ipynb                                      | efa7b034 (40.1 KB), b30f7733 (35.1 KB), ...               
  extractSum_vs_MkSlitFunc_vs_extractFromProfile.ipynb | d7eff341 (1.8 MB), 0fa29161 (1.8 MB), ...                 
  extract_and_extractFromProfile.ipynb                 | 038c8b83 (24.5 KB), edff9f8a (1.2 MB)                     
  paper.ipynb                                          | d808667e (91.4 KB)                                        
  rawspots.ipynb                                       | e9c02bab (564.4 KB), 26227920 (562.9 KB)                  
  test_tasks.ipynb                                     | 45e7909e (10.7 MB), e0d8c6fc (10.7 MB), ...               
  unittests.ipynb                                      | 0dbd739b (1.6 KB)                                         


In total, 278 object ids were changed. Full details are logged here:

  /Users/price/temp/obs_pfs.git.bfg-report/2016-11-28/16-02-02

BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive


Has the BFG saved you time?  Support the BFG on BountySource:  https://j.mp/fund-bfg


price@price-laptop:~/temp/obs_pfs.git (BARE:master) $ java -jar ~/Downloads/bfg-1.12.14.jar --delete-files "sampled*.fits"

Using repo : /Users/price/temp/obs_pfs.git

Found 58 objects to protect
Found 1 tag-pointing refs : refs/tags/3.1
Found 35 commit-pointing refs : HEAD, refs/heads/master, refs/heads/tickets/INFRA-36, ...

Protected commits
-----------------

These are your protected commits, and so their contents will NOT be altered:

 * commit 73d9dcce (protected by 'HEAD')

Cleaning
--------

Found 313 commits
Cleaning commits:       100% (313/313)
Cleaning commits completed in 313 ms.

Updating 1 Ref
--------------

  Ref                             Before     After   
  ---------------------------------------------------
  refs/heads/tickets/infra-0010 | 35873db1 | ff32fc22

Updating references:    100% (1/1)
...Ref update completed in 20 ms.

Commit Tree-Dirt History
------------------------

  Earliest                                              Latest
  |                                                          |
  ............Dmmmmmmmmm......................................

  D = dirty commits (file tree fixed)
  m = modified commits (commit message or parents changed)
  . = clean commits (no changes to file tree)

                          Before     After   
  -------------------------------------------
  First modified commit | 4c08bf11 | a3ac5ce8
  Last dirty commit     | a2f46825 | 21719eac

Deleted files
-------------

  Filename                     Git id            
  -----------------------------------------------
  sampledCombx2-IR-0-23.fits | 012ef06f (41.9 MB)
  sampledFlatx2-IR-0-23.fits | a0727dd1 (41.9 MB)


In total, 60 object ids were changed. Full details are logged here:

  /Users/price/temp/obs_pfs.git.bfg-report/2016-11-28/16-02-17

BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive


Has the BFG saved you time?  Support the BFG on BountySource:  https://j.mp/fund-bfg


price@price-laptop:~/temp/obs_pfs.git (BARE:master) $ java -jar ~/Downloads/bfg-1.12.14.jar --delete-files RedFiberPixels.fits.gz

Using repo : /Users/price/temp/obs_pfs.git

Found 58 objects to protect
Found 1 tag-pointing refs : refs/tags/3.1
Found 35 commit-pointing refs : HEAD, refs/heads/master, refs/heads/tickets/INFRA-36, ...

Protected commits
-----------------

These are your protected commits, and so their contents will NOT be altered:

 * commit 73d9dcce (protected by 'HEAD') - contains 1 dirty file : 
  - pfs/RedFiberPixels.fits.gz (18.1 MB)

WARNING: The dirty content above may be removed from other commits, but as
the *protected* commits still use it, it will STILL exist in your repository.

Details of protected dirty content have been recorded here :

/Users/price/temp/obs_pfs.git.bfg-report/2016-11-28/16-02-39/protected-dirt/

If you *really* want this content gone, make a manual commit that removes it,
and then run the BFG on a fresh copy of your repo.
       

Cleaning
--------

Found 313 commits
Cleaning commits:       100% (313/313)
Cleaning commits completed in 472 ms.

Updating 32 Refs
----------------

  Ref                                Before     After   
  ------------------------------------------------------
  refs/heads/master                | 73d9dcce | 710594f1
  refs/heads/tickets/INFRA-36      | e4011551 | b7b3c483
  refs/heads/tickets/INFRA-42      | e50ef116 | 61ac51d6
  refs/heads/tickets/INSTRM-3      | 2c69643a | 2bd4b6f1
  refs/heads/tickets/PIPE2D-108    | 9c253e69 | d3dd9e34
  refs/heads/tickets/PIPE2D-124    | 9999241b | 95ad708e
  refs/heads/tickets/PIPE2D-125    | 738fb2c0 | a788b4a7
  refs/heads/tickets/PIPE2D-128    | 61ee8c1e | 83d89a12
  refs/heads/tickets/PIPE2D-33     | f632e84e | fdf9dc86
  refs/heads/tickets/PIPE2D-35     | 80214484 | 5632a3d6
  refs/heads/tickets/PIPE2D-35-rhl | 696c7e52 | 58f4b53c
  refs/heads/tickets/PIPE2D-37     | 713ab1a5 | 63494793
  refs/heads/tickets/PIPE2D-38     | 3b221530 | a3b5bd15
  refs/heads/tickets/PIPE2D-43     | 2c568b72 | 3031f5c4
  refs/heads/tickets/PIPE2D-70     | 2ae06cc9 | d1577bb0
  ...

Updating references:    100% (32/32)
...Ref update completed in 110 ms.

Commit Tree-Dirt History
------------------------

  Earliest                                              Latest
  |                                                          |
  ......................DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD

  D = dirty commits (file tree fixed)
  m = modified commits (commit message or parents changed)
  . = clean commits (no changes to file tree)

                          Before     After   
  -------------------------------------------
  First modified commit | 830bc212 | 34103fbe
  Last dirty commit     | e09c0de8 | 1d7dd95f

Deleted files
-------------

  Filename                 Git id            
  -------------------------------------------
  RedFiberPixels.fits.gz | 2e6f5035 (18.1 MB)


In total, 393 object ids were changed. Full details are logged here:

  /Users/price/temp/obs_pfs.git.bfg-report/2016-11-28/16-02-39

BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive


Has the BFG saved you time?  Support the BFG on BountySource:  https://j.mp/fund-bfg


price@price-laptop:~/temp/obs_pfs.git (BARE:master) $ du -s
639M  .
price@price-laptop:~/temp/obs_pfs.git (BARE:master) $ git reflog expire --expire=now --all && git gc --prune=now --aggressive
Counting objects: 3005, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2430/2430), done.
Writing objects: 100% (3005/3005), done.
Total 3005 (delta 1532), reused 1128 (delta 0)
price@price-laptop:~/temp/obs_pfs.git (BARE:master) $ du -s
 65M  .
price@price-laptop:~/temp/obs_pfs.git (BARE:master) $ git remote add origin git@github.com:PaulPrice/obs_pfs.git
price@price-laptop:~/temp/obs_pfs.git (BARE:master) $ git push --mirror
Counting objects: 3005, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (898/898), done.
Writing objects: 100% (3005/3005), 64.87 MiB | 1.38 MiB/s, done.
Total 3005 (delta 1532), reused 3005 (delta 1532)
remote: Resolving deltas: 100% (1532/1532), done.
remote: warning: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
remote: warning: See http://git.io/iEPt8g for more information.
remote: warning: File pfs/redFiberTraces.fits is 62.85 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
To git@github.com:PaulPrice/obs_pfs.git
 * [new branch]      master -> master
 * [new branch]      tickets/INFRA-36 -> tickets/INFRA-36
 * [new branch]      tickets/INFRA-42 -> tickets/INFRA-42
 * [new branch]      tickets/INSTRM-3 -> tickets/INSTRM-3
 * [new branch]      tickets/PIPE2D-108 -> tickets/PIPE2D-108
 * [new branch]      tickets/PIPE2D-124 -> tickets/PIPE2D-124
 * [new branch]      tickets/PIPE2D-125 -> tickets/PIPE2D-125
 * [new branch]      tickets/PIPE2D-128 -> tickets/PIPE2D-128
 * [new branch]      tickets/PIPE2D-20 -> tickets/PIPE2D-20
 * [new branch]      tickets/PIPE2D-33 -> tickets/PIPE2D-33
 * [new branch]      tickets/PIPE2D-35 -> tickets/PIPE2D-35
 * [new branch]      tickets/PIPE2D-35-rhl -> tickets/PIPE2D-35-rhl
 * [new branch]      tickets/PIPE2D-37 -> tickets/PIPE2D-37
 * [new branch]      tickets/PIPE2D-38 -> tickets/PIPE2D-38
 * [new branch]      tickets/PIPE2D-43 -> tickets/PIPE2D-43
 * [new branch]      tickets/PIPE2D-70 -> tickets/PIPE2D-70
 * [new branch]      tickets/PIPE2D-77 -> tickets/PIPE2D-77
 * [new branch]      tickets/PIPE2D-80 -> tickets/PIPE2D-80
 * [new branch]      tickets/PIPE2D-83 -> tickets/PIPE2D-83
 * [new branch]      tickets/PIPE2D-90 -> tickets/PIPE2D-90
 * [new branch]      tickets/PIPE2D-92 -> tickets/PIPE2D-92
 * [new branch]      tickets/PIPE2D-98 -> tickets/PIPE2D-98
 * [new branch]      tickets/infra-0010 -> tickets/infra-0010
 * [new tag]         3.1 -> 3.1
 ! [remote rejected] refs/pull/1/head -> refs/pull/1/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/2/head -> refs/pull/2/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/2/merge -> refs/pull/2/merge (deny updating a hidden ref)
 ! [remote rejected] refs/pull/3/head -> refs/pull/3/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/4/head -> refs/pull/4/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/5/head -> refs/pull/5/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/6/head -> refs/pull/6/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/6/merge -> refs/pull/6/merge (deny updating a hidden ref)
 ! [remote rejected] refs/pull/7/head -> refs/pull/7/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/8/head -> refs/pull/8/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/8/merge -> refs/pull/8/merge (deny updating a hidden ref)
error: failed to push some refs to 'git@github.com:PaulPrice/obs_pfs.git'

Comment by aritter [ 03/Dec/16 ]

Looked at the comments and couldn't find any obvious problems. Tests and runPipeline script pass. No objections to merge

Comment by price [ 06/Dec/16 ]

Thanks Andreas.

Pushed to the main GitHub site and verified the reduced size. Sent a heads-up e-mail to pfs_software.

[INFRA-53] Cleanup obs_pfs in git Created: 11/Aug/16 Updated: 06/Dec/16 Resolved: 06/Dec/16
Status:	Done
Project:	Software Development Infrastructure
Component/s:	None
Affects Version/s:	None
Fix Version/s:	None

[INFRA-53] Cleanup obs_pfs in git Created: 11/Aug/16 Updated: 06/Dec/16 Resolved: 06/Dec/16