https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-292 DRP 2-D Pipeline <p>Our dependency tree could be stripped down, simplifying and speeding up the install.</p> <p>We can dump our dependency on obs_subaru by using the <tt>CrosstalkTask</tt> in ip_isr.<br/> We could remove our dependency on pipe_drivers and pipe_tasks by forking <tt>constructCalibs.py</tt>, <tt>ingestImages.py</tt> and <tt>ingestCalibs.py</tt>.</p> PIPE2D-292

Strip down dependency tree

Story Normal Done Done price price Tue, 25 Sep 2018 20:27:58 +0000 Tue, 18 Dec 2018 19:31:39 +0000 Tue, 18 Dec 2018 19:31:39 +0000 0 2 <p>We also need <tt>RepairTask</tt> from pipe_tasks.</p> <p>OK, we needed a few other itty bitty files from pipe_tasks and pipe_drivers (a total of about 10 functional python files, plus some bin scripts), but I think it's quite manageable. I propose to call the new repo drp_lsst; it's currently on <a href="https://github.com/PaulPrice/drp_lsst" class="external-link" rel="nofollow">my GitHub</a> until we can create a place for it in the PFS GitHub.</p> <p>Because we no longer want to build all of LSST and we don't want to use the full LSST docker that's provided by LSST, I've reworked the installation scripts. We now install just ip_isr, meas_algorithms and ctrl_pool as top-level products.</p> <p>I also took this opportunity to refactor the Dockerfiles. They have been in the top-level directory of pfs_pipe2d, mixed in with everything else, and it wasn't clear how to build them. I've moved them into their own directory, added a <tt>Makefile</tt>, and renamed them to be more consistent with their names as on DockerHub. There's a new <tt>pfs_lsst</tt> image that will replace what we were using from LSST. The pfs_pipe2d image is now about 45% smaller than it used to be. The integration test in Travis is now run through the docker <tt>Makefile</tt> (<tt>make test</tt>), as that can both build the image and run the test.</p> <p>Unrelated, but because I was thinking about the reduction in time to run the integration test in Travis, I deleted the old sim data in drp_stella_data: it's no longer used now we have real data. However, I haven't seen a speedup in Travis because the integration test first clones the <tt>master</tt> branch before checking out the ticket branch, which means it's still downloading everything until we merge the ticket branch.</p> <p>Please do not do this.</p> <p>We are going to want to move to PipelineTask and the new butler when they become available, so the dependency on pipe_tasks etc. must remain.</p> <p>We can remove things above the level we use, or packages that we never import even indirectly.</p> <p>OK, keeping pipe_tasks and pipe_drivers is not as efficient as the previous method for stripping the dependencies (76 packages instead of about 57, and 3.84 GB instead of 3.24 GB), but it does clean some things up. The changes to Docker stand.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a>, would you please review these changes?</p> <p>Merged to master.</p> Development Rank 0|s0014g: Reviewers cloomis