https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/INSTRM-790 Instrument control development <p>This is not new, but it often happened this time</p> <p>it can happen at the ion pumps start as shown below or in operation by the monitoring.</p> <p>it would be "nice" to have a way to handle that before starting operation at Subaru</p> <p> </p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2019-10-03 15:23:19 xcu_r1 w text="failed to create connect or send to ion pump: [Errno 111] Connection refused" 2019-10-03 15:23:19 xcu_r1 f text="command failed: ConnectionRefusedError(111, 'Connection refused') in sendOneCommand() at /software/mhs/products/Linux64/ics_xcuActor/1.11.1/python/xcuActor/Controllers/ionpump.py:65" </pre> </div></div> <p> </p> <p> </p> <p> </p> INSTRM-790

ion pumps communication issue

Bug Normal Open Unresolved cloomis fmadec SPS Mon, 7 Oct 2019 08:09:50 +0000 Wed, 16 Oct 2019 15:10:54 +0000 ics_xcuActor 0 2 <p>The chained pair of ionpump controllers which controls all six ion pumps for the three cryostats are accessible as two RS-485 nodes behind a single MOXA RS-485 port. This cluster of things is commanded from three independent actors. This cannot be reliable, really. One of the earlier tickets discussing ion pump communications problems suggested having either an <tt>ionpumpActor</tt> or at least a single program to sequence all commands to the controllers. The traffic is very simple, so either seems rational. Will think about which is best.</p> <p>I looked more carefully at the MOXA configuration options, and do not believe we can use it to sequence TCP connections: MOXAs can allow multiple <em>concurrent</em> connections, but that would be <b>much</b> worse.</p> <p>An inventory of all the <tt>Connection refused</tt> failures in 2019 does suggest that simply retrying after some small random delay would significantly improve matters. But that cannot entirely fix the real problem.</p> <p>For r1 and b1 together, 2019 showed 1_260_000 commands with all but 49 being <tt>status</tt> commands; 11_000 of those connections were refused. All but 136 of the refused connections were from when the periodic status queries for the two cryostats synced up (07-1* and 09-1*). The periodic tasks are re-scheduled based on the system clock and not from the end of a command, so it is very easy to get locked into a hopeless failure. Different ticket, but naively retrying to fix this ticket's problems would probably make things worse without addressing that. I'd rather add a new actor than change that.</p> Relates INSTRM-791 Development Rank 0|02qpuc:i6001gs8 Sprint SM1-2019 P