[PIPE2D-598] Upgrade base LSST stack to latest version Created: 06/Jun/20  Updated: 26/May/22  Resolved: 26/May/22

Status: Done
Project: DRP 2-D Pipeline
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Normal
Reporter: hassan Assignee: price
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Blocks
blocks PIPE2D-506 Migrate code to use LSST gen3 middleware Done
Story Points: 2
Sprint: 2DDRP-2022 B, 2DDRP-2022 C, 2DDRP-2022 D
Reviewers: hassan

 Description   

Please update the base LSST stack version (currently 18.1.0) to the latest version.

It has been a while since an upgrade has taken place. We need to align ourselves to the latest version to ensure support from the Rubin team, and we need to make use of the latest gen3 middleware features.

 Comments   
Comment by price [ 25/Feb/22 ]

Just about ready to roll this out, but I should point out the following:

There are lots of warnings when we run scons. Not much we can do about them yet.

  • Deprecation warnings from pytest about pytest_flake8 will be gone when we upgrade to 23.0.2.
  • Could not find EUPS product dir is fixed on main, but prolly won't be backported.
  • Numpy binary incompatibility warnings appear to be normal; people freak out about them, but haven't been able to determine the origin and it hasn't caused any problems.

The butler produces a deprecation warning about Gen2 very frequently. Let's set the environment variable:

PYTHONWARNINGS="ignore:Gen2 Butler has been deprecated:FutureWarning:"
Comment by price [ 25/Feb/22 ]

I won't merge this until the group as a whole has approved the upgrade, but the changes to our packages are ready for review.

The integration test and the weekly have both passed with the upgrade. I'm in the process of building a new stack on tiger and new Docker images. I also need to prepare an email announcing the upgrade with directions on how to build a new stack.

Comment by price [ 25/Feb/22 ]

The new stack has been installed on tiger in /projects/HSC/PFS/stack/20220224.

Once this merges, the new stack can be installed following the installation instructions in the pipeline user guide.

Comment by hassan [ 19/Mar/22 ]

Integration looks fine and reviewed all pull requests.

However, it has been agreed that others at Princeton should evaluate this before merging to master, so will leave the ticket status as 'In Review' for the time being.

Comment by price [ 29/Mar/22 ]

Please note that there is a new procedure that needs to be run on calibs to ingest the camera defects:

ingestCuratedCalibs.py /projects/HSC/PFS/Subaru --calib /path/to/CALIB $DRP_PFS_DATA_DIR/curated/pfs/defects

I've just updated the writable Subaru calibs on tiger:

for calib in CALIB-* ; do ingestCuratedCalibs.py /projects/HSC/PFS/Subaru --calib $calib $DRP_PFS_DATA_DIR/curated/pfs/defects ; done

Other calib collections (anything I've missed at Princeton, and at other sites) will also need to be updated.

Comment by price [ 03/May/22 ]

This has been more or less superseded by PIPE2D-506.

Comment by price [ 26/May/22 ]

Merged.

Generated at Sat Feb 10 15:55:15 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.