https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/INSTRM-1047 Instrument control development <p>(This is almost a verbatim copy of an email I sent to Craig and Robert some time ago; at that time I couldn't open issues on JIRA. Please excuse me if I'm filing this in the wrong category ... I'm not yet completely familiar with the internal organization...)</p> <p>The packages ets_fiberalloc and ets_shuffle are going to depend on pfs_utils for coordinate conversions in the future. In preparation for that I tried to install the package locally. This requires scons, which was not a problem to install.<br/> It also depends in lsst.sconsUtils, which I cloned from Github. Here the trouble started. The "scons" command for lsst.sconsUtils insists on calling "pytest" (hardwired), which is called "pytest-3" on my machine. I managed to adjust this, but now I get</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> running global pytest... ERROR: usage: pytest-3 [options] [file_or_dir] [file_or_dir] [...] pytest-3: error: unrecognized arguments: --session2file=tests/.tests/pytest-sconsUtils.xml.out inifile: None rootdir: /home/martin/codes/sconsUtils </pre> </div></div> <p>Do you have any advice how to proceed?</p> INSTRM-1047

Problems installing pfs_utils package

Bug Normal Done Done Unassigned Martin Reinecke Sat, 8 Aug 2020 09:04:27 +0000 Tue, 7 Sep 2021 14:58:36 +0000 Tue, 7 Sep 2021 14:58:36 +0000 pfs_utils 0 5 <p>So what do we do about this PITA? As a reminder, <tt>pfs_utils</tt> is the package dedicated to code shared between the ICS and DRP worlds (fiber id mappings, core coordinate transforms, etc), and was supposed to not require any <em>other</em> ICS or DRP packages. </p> <p>But for DRP (scons) installation and CI testing, <tt>sconsUtils</tt> has been added, which drags in a bunch of DRP infrastructure. Instead of being a trivial to install standalone package, it forces everyone to install some part of the Rubin universe. Subaru ICS, MPA, ASRD are the </p> <p>So pull sconsUtils out? Add it and its friends as a requirement? The LSST requirement on some utterly antique <tt>pytest</tt> and thus <tt>attrs</tt> has already given us (ICS) trouble; with my ICS hat on I am very disinclined to buy into any more dependencies.</p> <p>If it helps, this is not really urgent on my side. I currently just copy the Python files to a place where the interpreter finds them.</p> <p>Perhaps it would already be sufficient to add a very basic <tt>setup.py</tt>.</p> <p>Agreed on adding basic setup.py. </p> <p>I'm encountering the same problem with "datamodel". It would be really great if this issue could be addressed for all concerned packages.</p> <p>In the meantime I'll have to copy Python files around by hand...</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=Martin.Reinecke" class="user-hover" rel="Martin.Reinecke">Martin Reinecke</a>: Can you provide the most recent steps you made, and a copy of the error message you saw please?</p> <p>It's not so much a matter of commands failing, but rather of no well-defined installation procedure existing.</p> <p>As I mentioned in the first post, I'm not aware of any document describing how these packages should be installed. I tried finding out by myself, but it requires manual installation of a not very common build tool (scons), then additionally cloning lsst.sconsUtils (also manually). Last time I tried, the package didn't install even then, but that's not really the essence of this issue.</p> <p>The proposed solution (add a simple setup.py script) seems to have general agreement, but it's not implemented yet.</p> <p> </p> <p>I will back off the setup.py suggestion and push for using requiring basic <tt>eups</tt> instead. Basically, all the PFS packages (both ICS and DRP) use a package directory tree which does not work well with the python setuptool/distutils/pkg_resources policies. Specifically, we expect a top-level directory with <tt>bin/, etc/, docs/, python/pfs/utils/, data/, examples/</tt>, etc. in it, and use our own mechanism to adjust PYTHONPATH and PATH to match. The builtin tools put the <em>content</em> of our <tt>python</tt> directory into <tt>...../site-packages/</tt>, merging all pfs packages into that, and leave PYTHONPATH alone. Extra files and directories are put in other, non-site-packages directories.</p> <p>This matters for pfs_utils, which includes some stable but essential data files in <tt>data/fiberids/</tt>, and which are found by using <tt>eups</tt> routines.</p> <p>One could certainly argue that what we do is not what one would start with in 2020, but we started much earlier and this is what we have. We can semi-cheat and get something close to what PFS does, but not without depending on undocumented behavior. Using bare-bones eups ("setup -r .") seems like a not-insane middle ground.</p> <p>OK, so <tt>eups</tt>. I think there are some basic intro docs both in the Rubin/HSC and the PFS worlds. Someone might have a decent pointer....</p> <p>I have added a skeleton `setup.py` file on the tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/INSTRM-1047" title="Problems installing pfs_utils package" class="issue-link" data-issue-key="INSTRM-1047"><del>INSTRM-1047</del></a> branch of the datamodel repo, which does the trick for me. I also see that <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a> has been working on this for pfs_utils, but I don't manage to get his solution to work for me yet.</p> <p>If wanted, I can provide a simple setup.py script for pfs_utils as well. Please let me know!</p> <p>I took the liberty of pushing a setup.py file to the tickets/<a href="https://pfspipe.ipmu.jp/jira/browse/INSTRM-1047" title="Problems installing pfs_utils package" class="issue-link" data-issue-key="INSTRM-1047"><del>INSTRM-1047</del></a> branch of the pfs_utils repository. Please feel free to remove it again if a different sort of solution is desired!</p> <p>What still needs to happen to close this ticket and merge the changes?</p> <p>I merged datamodel and pfs_utils.</p> <p>Thank you!</p> <p>Can we close this ticket?</p> <p>I'd say yes. If anything else turns up, we can open another one.</p> <p>Do I have to do special Jira magic so that it can be closed?</p> Development Rank 0|zzs2fk: