[INSTRM-306] Define python environment Created: 22/Mar/18  Updated: 09/Nov/22  Resolved: 22/Mar/18

Status: Open
Project: Instrument control development
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Normal
Reporter: cloomis Assignee: cloomis
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates
relates to INSTRM-1615 RFC: Consider basing ICS on rubin-env... Open

 Description   

We should define which python version, distribution, and modules we expect to have installed.

Even though most modern distribution systems should be fine I suggest that we only support miniconda+products for now, just to minimize the unknowns. But if anyone want to use apt/pip/homebrew and has the following versions it should be fine. Full anaconda 5.2+ distributions should work, though several modules will need to be added.

The MHS products currently require the following python modules. Obviously not all products require all modules; I'll state the main purpose/requirement for each. The listed versions are usually whatever gets installed by conda install on 2018-03-20; in the cases where I know that the version matters I'll add an exclamation point.

In general there will be version conflicts, since we are installing individual modules. So this is more of a guide than a prescription. All versions are expected to be upgradable.

Nearly all modules include sub-modules dependencies, which are not listed.

The absolutely essential packages for all actors are:

  • python: 3.6! – asyncio is stable and supported from 3.6
  • twisted: 17.9 – for tron_actorcore
  • ply: 3.11! – for all tron_actorcore parsing

On top of those, various packages require sets of the following:

  • dnspython: 1.16 - for SPS actors, at least.
  • numpy: 1.14 – for the detector readouts
  • cython: 0.27! - for the detector readouts, several others.
  • pyserial: 3.4 – for many device controlling actors, even non-serial ones.
  • future: 0.16! – only necessary for code upgraded from python2
  • astropy: 3.0.1 – import astropy.io.fits as pyfits
  • fitsio: 0.9.11! (from pip) – pyfits can be too inefficient for the H4 ramps or on small machines.
  • ruaml_yaml: 0.15.35 – ics_config, others.
    This is a painful choice: pyyaml is moribund, ruaml_yamel is on bitbucket and has a new API. Not obvious.
  • psycopg2-binary: 2.7.4 (from pip) – This has switched to a slightly confusing "binary" install.
  • qt: 5.9.5 – for any pyqt GUIs with actorcore parts
  • pyqt: 5.9.5 – same
  • qt5reactor: 0.5 (from pip) – for any pyqt GUIs with actorcore parts
  • matplotlib: 2.2 – for various GUIs.
  • fysom: 2.1.5 – for FSMs in tron_actorcore
  • sqlalchemy: 1.2.10 – for ics_archiver, any querying code (ics_sps_engineering_stuff, etc.
  • gitpython: 2.1.11 – for the version keys.
  • redis-py: 3.5.3 – for gen2 status streaming. Also underlying redis pakage
  • msgpack: 1.0.2 (from pip) – for gen2 status streaming
  • pymongo: 3.11.3 (from pip) – for gen2 status streaming
  • ginga : 3.10 (from pip) - for real time displau

The following have been added to the root environment but should perhaps have been put into an different one. Jupyter and opencv have impressive dependency lists.

  • jupyter: 1.0.0 – for notebooks
  • scipy: 1.1.0 – just because people do use it.
  1. - opencv: 3.4.1 – for MCS visualization, at least.


 Comments   
Comment by Yoshida, Hiroshige [ 22/Mar/18 ]

psycopg2 is required by ics_archiver.

Comment by shimono [ 02/May/18 ]

cloomis what mean by "!" for version number?
Yoshida, Hiroshige need both psycopg2-binary and psycopg2?

Comment by Yoshida, Hiroshige [ 02/May/18 ]

Both conda and pip now have a package called "psycopg2". So only "psycopg2" by conda?

Comment by hassan [ 03/May/18 ]

Hi @shimono regarding the "!": Craig mentioned in the description that

"[...] in the cases where I know that the version matters I'll add an exclamation point."

so it refers to when that version number is important.

 

Comment by cloomis [ 03/May/18 ]

psycopg2 is deprecated, and is being replaced by psycopg2-binary. At least that's the way I read it. You certainly do not want both.

Comment by shimono [ 03/May/18 ]

hassan i know, but not sure for lower values,,, like sys.version could be wrote as sys.version = (3, 6) or sys.version >= (3,6). (or (3,6,0) would not be the case, i suppose...)

Comment by hassan [ 03/May/18 ]

I see shimono - thanks for the clarification! 

Comment by Yoshida, Hiroshige [ 23/Aug/18 ]

fysom is required for FSM in tron_actorcore

Comment by Yoshida, Hiroshige [ 23/Aug/18 ]

FYI ruaml.yaml (not ruaml_yaml) >=0.15.52 are broken for me

Comment by cloomis [ 23/Aug/18 ]

How, specifically?

Comment by Yoshida, Hiroshige [ 23/Aug/18 ]

I get this error:

Traceback (most recent call last):
File "/home/hyoshida/src/pfs-sbr-id/qt+ipc/er_flow_monitor_gui.py", line 30, in update_status
self.ui.lineEdit_1_1.setText('%.0f' % data.readings[config.electronics_rack.manifold.intake_flow_rate.channel - 1])
AttributeError: 'CommentedMap' object has no attribute 'electronics_rack'
Aborted

YAML files looks like this

electronics_rack:
    manifold:
        intake_flow_rate:
            device: keys2
            channel: 29
        intake_temperature:
            device: keys2
            channel: 30

Or, am I violating any YAML spec.???

Comment by Yoshida, Hiroshige [ 23/Aug/18 ]

Actually, it may not be an issue with ruamel.yaml. I am "munchifying" config as:

config = munch.munchify(ruamel.yaml.YAML().load(open(args.config_file, 'r')))

Then I am getting the error above. However, the process round-trips and recreate a correct input config:

ruamel.yaml.YAML().dump(munch.unmunchify(config), sys.stdout)

ruamel.yaml.YAML().load() used to return a CommentedMap object; now it returns ordereddict

Was off-topic - please ignore my last two comments

Generated at Sat Feb 10 16:23:38 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.