https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/INSTRM-878 Instrument control development <p>tron crashed during the night (3:54am UTC on Jan 13 2020) .</p> <p>after a restart of tron, some actors(xcu b1 and r1, enu_sm1) were not able to reconnect to Tron, but some were :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 2020-01-13T15:46:28.182 sent hub status 2020-01-13T15:46:28.186 hub i version="need_to_read_git_version" 2020-01-13T15:46:28.189 hub i actors=hub,keys,msg,gen2,spsait,seqno,meb,sps 2020-01-13T15:46:28.190 hub i commanders=client.v1,client.v3,client.v4,client.v5,client.v8,client.v13,client.</pre> </div></div> <p> </p> <p>the error message is</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 2020-01-13 15:46:13.997Z cmdr 30 CmdrConnection.py:79 in doStart 2020-01-13 15:46:14.001Z cmdr 30 CmdrConnection.py:108 CmdrConnection failed: [Failure instance: Traceback (failure with no frames): &lt;class 'twisted.internet.error.ConnectionRefusedError'&gt;: Connection was refused by other side: 111: Connection refused.</pre> </div></div> <p> </p> <p>I rebooted the bee_r1 but got the same error.</p> <p>Arnaud tried to start enu_sm1 on another machine (shell-ics) and it worked... but it doesn't work on rack5 while it is working for rough on the same machine...</p> <p>one thing strange is that the command oneCmd.py hub status doesn't work on rack5 and bee-r1.</p> <p>So we do not have the current status of the 2 cameras... which are supposed to pumpdown (r1 gatevalve certainly close during the reboot).</p> <p> </p> INSTRM-878

actors cannot connect to tron

Bug Normal Done Done arnaud.lefur fmadec SM1 SPS Mon, 13 Jan 2020 15:54:20 +0000 Wed, 19 Feb 2020 14:51:48 +0000 Wed, 12 Feb 2020 22:33:50 +0000 ics_xcuActor 0 4 <p>Reconnections failed about half the time because I had mistakenly added a second IP address for <tt>tron</tt>, regardless of which host the connections were made from. That should now be fixed, so I will drop the Blocker level. The crash may or may not be related; I'll look.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=arnaud.lefur" class="user-hover" rel="arnaud.lefur">arnaud.lefur</a> will try to reproduce the problem at LAM. Ticket will be handed back to Craig on his return.</p> <p>Wait &#8211; is there any point in working on this? There were two IP addresses for the hostname 'tron'. Actors (or oneCmd.py) which resolved to the bad one could not connect. A 50/50 chance of a connection working.</p> <p>Or are you trying to make tron crash? That could be interesting, I suppose. But given the history surely there are more important issues to work on, no? </p> <p>yes, I was thinking about the crash itself, I'm nearly sure that <a href="https://pfspipe.ipmu.jp/jira/browse/INSTRM-879" title="stsBuffer get never cleared and grow up in an uncontrolled manner. " class="issue-link" data-issue-key="INSTRM-879"><del>INSTRM-879</del></a> was the root cause.</p> <p>But I can run some test in parallel, it won't take much time.</p> <p>With <a href="https://pfspipe.ipmu.jp/jira/browse/INSTRM-879" title="stsBuffer get never cleared and grow up in an uncontrolled manner. " class="issue-link" data-issue-key="INSTRM-879"><del>INSTRM-879</del></a>, logfile has grown very fast, basically :  data with a sample rate of 15sec  for 30 days gives : 30 * 24 * 60 * 4 = 172800 samples</p> <p>with this bug log size would reach about n * (n+1)/2 = 1.5e10 rows </p> <p>with only one data and we have about 30, I bet that was the issue !</p> <p>There are three problems, and we fixed two:</p> <ul class="alternate" type="square"> <li>the tron server <em>probably</em> crashed due to a huge flood of traffic from the alerts actor. The tron crash has not been fixed.</li> <li>the alertsActor was flooding tron. That bug has been fixed.</li> <li>the dns configuration at subaru has two addresses for the tron host, so connections failed about half the time. This has been fixed.</li> </ul> <p>We could open a new ticket asking tron to be more resilient to denial-of-service attacks. </p> Relates INSTRM-879 Development Rank 0|02qpuc:i6001gtpi8zx4 Sprint SM1PD-2020 A SM1PD-2020 B Story Points 1.0