https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-598 DRP 2-D Pipeline <p>Please update the base LSST stack version (currently 18.1.0) to the latest version.</p> <p>It has been a while since an upgrade has taken place. We need to align ourselves to the latest version to ensure support from the Rubin team, and we need to make use of the latest gen3 middleware features.</p> PIPE2D-598

Upgrade base LSST stack to latest version

Task Normal Done Done price hassan Fri, 5 Jun 2020 17:13:08 +0000 Wed, 25 May 2022 19:11:30 +0000 Wed, 25 May 2022 19:11:30 +0000 0 2 <p>Just about ready to roll this out, but I should point out the following:</p> <p>There are lots of warnings when we run scons. Not much we can do about them yet.</p> <ul> <li>Deprecation warnings from pytest about <tt>pytest_flake8</tt> will be gone when we upgrade to 23.0.2.</li> <li>Could not find EUPS product dir is fixed on main, but prolly won't be backported.</li> <li>Numpy binary incompatibility warnings appear to be normal; people freak out about them, but haven't been able to determine the origin and it hasn't caused any problems.</li> </ul> <p>The butler produces a deprecation warning about Gen2 very frequently. Let's set the environment variable:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> PYTHONWARNINGS=<span class="code-quote">"ignore:Gen2 Butler has been deprecated:FutureWarning:"</span> </pre> </div></div> <p>I won't merge this until the group as a whole has approved the upgrade, but the changes to our packages are ready for review.</p> <p>The integration test and the weekly have both passed with the upgrade. I'm in the process of building a new stack on tiger and new Docker images. I also need to prepare an email announcing the upgrade with directions on how to build a new stack.</p> <p>The new stack has been installed on tiger in <tt>/projects/HSC/PFS/stack/20220224</tt>.</p> <p>Once this merges, the new stack can be installed following the installation instructions in the pipeline user guide.</p> <p>Integration looks fine and reviewed all pull requests.</p> <p>However, it has been agreed that others at Princeton should evaluate this before merging to master, so will leave the ticket status as 'In Review' for the time being.</p> <p>Please note that there is a new procedure that needs to be run on calibs to ingest the camera defects:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ingestCuratedCalibs.py /projects/HSC/PFS/Subaru --calib /path/to/CALIB $DRP_PFS_DATA_DIR/curated/pfs/defects </pre> </div></div> <p>I've just updated the writable Subaru calibs on tiger:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">for</span> calib in CALIB-* ; <span class="code-keyword">do</span> ingestCuratedCalibs.py /projects/HSC/PFS/Subaru --calib $calib $DRP_PFS_DATA_DIR/curated/pfs/defects ; done </pre> </div></div> <p>Other calib collections (anything I've missed at Princeton, and at other sites) will also need to be updated.</p> <p>This has been more or less superseded by <a href="https://pfspipe.ipmu.jp/jira/browse/PIPE2D-506" title="Migrate code to use LSST gen3 middleware" class="issue-link" data-issue-key="PIPE2D-506"><del>PIPE2D-506</del></a>.</p> <p>Merged.</p> Blocks PIPE2D-506 Development Rank 0|02qpjd:0c Reviewers hassan Sprint 2DDRP-2022 B 2DDRP-2022 C 2DDRP-2022 D Story Points 2.0