[PIPE2D-292] Strip down dependency tree Created: 26/Sep/18 Updated: 19/Dec/18 Resolved: 19/Dec/18 |
|
| Status: | Done |
| Project: | DRP 2-D Pipeline |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Story | Priority: | Normal |
| Reporter: | price | Assignee: | price |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Reviewers: | cloomis |
| Description |
|
Our dependency tree could be stripped down, simplifying and speeding up the install. We can dump our dependency on obs_subaru by using the CrosstalkTask in ip_isr. |
| Comments |
| Comment by price [ 04/Oct/18 ] |
|
We also need RepairTask from pipe_tasks. |
| Comment by price [ 06/Oct/18 ] |
|
OK, we needed a few other itty bitty files from pipe_tasks and pipe_drivers (a total of about 10 functional python files, plus some bin scripts), but I think it's quite manageable. I propose to call the new repo drp_lsst; it's currently on my GitHub until we can create a place for it in the PFS GitHub. Because we no longer want to build all of LSST and we don't want to use the full LSST docker that's provided by LSST, I've reworked the installation scripts. We now install just ip_isr, meas_algorithms and ctrl_pool as top-level products. I also took this opportunity to refactor the Dockerfiles. They have been in the top-level directory of pfs_pipe2d, mixed in with everything else, and it wasn't clear how to build them. I've moved them into their own directory, added a Makefile, and renamed them to be more consistent with their names as on DockerHub. There's a new pfs_lsst image that will replace what we were using from LSST. The pfs_pipe2d image is now about 45% smaller than it used to be. The integration test in Travis is now run through the docker Makefile (make test), as that can both build the image and run the test. Unrelated, but because I was thinking about the reduction in time to run the integration test in Travis, I deleted the old sim data in drp_stella_data: it's no longer used now we have real data. However, I haven't seen a speedup in Travis because the integration test first clones the master branch before checking out the ticket branch, which means it's still downloading everything until we merge the ticket branch. |
| Comment by rhl [ 06/Oct/18 ] |
|
Please do not do this. We are going to want to move to PipelineTask and the new butler when they become available, so the dependency on pipe_tasks etc. must remain. We can remove things above the level we use, or packages that we never import even indirectly. |
| Comment by price [ 06/Oct/18 ] |
|
OK, keeping pipe_tasks and pipe_drivers is not as efficient as the previous method for stripping the dependencies (76 packages instead of about 57, and 3.84 GB instead of 3.24 GB), but it does clean some things up. The changes to Docker stand. cloomis, would you please review these changes? |
| Comment by price [ 19/Dec/18 ] |
|
Merged to master. |