[INSTRM-1615] RFC: Consider basing ICS on rubin-env conda/python environment Created: 01/Jun/22 Updated: 11/Nov/22 |
|
| Status: | Open |
| Project: | Instrument control development |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Task | Priority: | Normal |
| Reporter: | cloomis | Assignee: | Unassigned |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Description |
|
The production ICS python environment is currently a conda 3.7 + handful-of-packages thing, as defined in INSTRM-306 The hxActor requires 3.8, because of how nice and reliable the new multiprocessing.shared_memory module was. I have been testing for months with 3.9, figuring that if we update we might as well catch up with the latest stable version. Rubin has switched to a conda-forge based environment named rubin-env. For the version of DRP that PFS has just switched to, that is rubin-env 3.0.0 (python 3.8). Their current environment is 4.0.0, which is python 3.10 (the current stable version). Both of those have current cython, astropy, fitsio, psycopg2, etc, and in general are up-to-date, very very much unlike the previous LSST environment. Promisingly, the hxActor and oneCmd are happily running under something like the following at JHU: mamba create -n rubin3_ics -c conda-forge rubin-env=3.0.0 conda activate rubin3_ics mamba install -c conda-forge twisted ply dnspython astroplan EUPS_PATH=$EUPS_PATH:/software/mhs/products [ there will be a few more packages needed. pyqt, qt5reactor, fysom (still pip only, grr), etc. Plus Jupyter, etc. Does conda activate --stack work? ] So, we certainly could base ICS on the Rubin environment. Would that be a good idea? I don't quite know yet: I have been trampled while dancing with this particular elephant before. But I think we can and should look into it. |
| Comments |
| Comment by price [ 02/Jun/22 ] |
|
The 2D pipeline has just upgraded the base LSST version and adopted their environment because it's convenient, not because I don't have lingering concerns. Foremost of those concerns is that rubin-env leaves many packages un-pinned (compare the list of pins with the list of packages), which means that different installations can produce different results when using the same pipeline version. |
| Comment by cloomis [ 05/Aug/22 ] |
|
For ICS, the python 3.10 rubin-env=4.1.0 environment seems to be basically fine. I have only tested a couple of actors, and have not restarted the run hub or the archiver, but do not expect trouble there either. To be specific: mamba create -n rubin4_ics -c conda-forge rubin-env=4.1.0 conda activate rubin4_ics mamba install -c conda-forge twisted ply dnspython astroplan pyqt qt5reactor ipython pyserial pip install fysom In rubin-env 4.1.0, at least, both numpy and |
| Comment by cloomis [ 11/Nov/22 ] |
|
I liked having INSTRM-306 as a live reference of the accumulated requirements for condo-ics, and we should make one for rubin3-ics. The above stanza is a start, and there have been a few more packages added. |