https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/INSTRM-594 Instrument control development <p>The ion pump controller at JHU just coughed out a channel, where the voltage dropped to 3000V, current &amp; pressure dropped to 0, but the controller did not report any errors. Moving the pump to a different channel made it work again.</p> <p>I'm sure we'd spot this from the pressure and current being 0, but we should probably be extra careful.</p> INSTRM-594

Alert on voltage/current anomalies in ion pump controllers

Task Normal Done Done arnaud.lefur cloomis SPS Fri, 11 Jan 2019 16:24:19 +0000 Tue, 10 Mar 2020 20:48:28 +0000 Tue, 10 Mar 2020 09:54:21 +0000 ics_alertsActor 0 4 <p>Damn. Happened again, on another channel (we earlier moved pump2 from channel 2 to 3, and just moved pump1 from 1 channel 1 to 4).</p> <p>In any case, no errors. Only invalid because the commanded state is 1 but the current and pressure are 0.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2019-09-04 19:12:19.712Z cmds 20 CommandLink.py:121 &gt; 1 3308 i ionPump1=1,7000,0.0018,35, 3.8e-06 2019-09-04 19:12:19.714Z cmds 20 CommandLink.py:121 &gt; 1 3308 i ionPump1Errors=0x00,OK 2019-09-04 19:12:49.708Z cmds 20 CommandLink.py:121 &gt; 1 3317 i ionPump1=1,7000,0.0015,34, 3.1e-06 2019-09-04 19:12:49.711Z cmds 20 CommandLink.py:121 &gt; 1 3317 i ionPump1Errors=0x00,OK 2019-09-04 19:13:19.655Z cmds 20 CommandLink.py:121 &gt; 1 3326 i ionPump1=1,7000,0.0013,34, 2.5e-06 2019-09-04 19:13:19.658Z cmds 20 CommandLink.py:121 &gt; 1 3326 i ionPump1Errors=0x00,OK 2019-09-04 19:13:49.797Z cmds 20 CommandLink.py:121 &gt; 1 3335 i ionPump1=1,7000,0.001,34, 1.7e-06 2019-09-04 19:13:49.800Z cmds 20 CommandLink.py:121 &gt; 1 3335 i ionPump1Errors=0x00,OK 2019-09-04 19:14:19.712Z cmds 20 CommandLink.py:121 &gt; 1 3344 i ionPump1=1,3100,0,34, 0 2019-09-04 19:14:19.715Z cmds 20 CommandLink.py:121 &gt; 1 3344 i ionPump1Errors=0x00,OK 2019-09-04 19:14:49.705Z cmds 20 CommandLink.py:121 &gt; 1 3353 i ionPump1=1,3100,0,33, 0 2019-09-04 19:14:49.708Z cmds 20 CommandLink.py:121 &gt; 1 3353 i ionPump1Errors=0x00,OK 2019-09-04 19:15:03.001Z cmds 20 CommandLink.py:121 &gt; 2 5327 i ionPump1=1,3100,0,33, 0 2019-09-04 19:15:03.002Z cmds 20 CommandLink.py:121 &gt; 2 5327 i ionPump1Errors=0x00,OK </pre> </div></div> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=arnaud.lefur" class="user-hover" rel="arnaud.lefur">arnaud.lefur</a> and I were just talking about issues like this in the alertActor, so I'll flesh this out a bit.</p> <p>This is an example where simple <tt>alertsActor</tt> limits are not sufficient. We can either add alertsActor logic or ionpump controller logic. </p> <p>Inside the alertsActor I think the way do do this would be to set a code callback on <tt>ionPump1Errors</tt>, and if it is 0x00/OK look at the <tt>ionPump1</tt> keyword and check for state == 1 and pressure == 0. For slightly disgusting reasons: if the error keyword is not 0x00 we should use that, and the errors key comes just <em>after</em> the main keyword, so any logic has access to both keys.</p> <p>FWIW there is no "ionpump error" STS datum, but we should make one, based on the numeric field. Also, the ionpump error key should grow a descriptive string field, as we have for the coolers and gate valve, etc.</p> <p>But I think the fix should actually be made in the xcuActor, which should "create" an error bit in this instance.</p> <p>I think we can just use the second (string) ��field as the alert description. So I guess create a STRING-only STS datum and use an (inverted) regex alert against "OK"? In any case this is back to being just an alertsActor ticket.</p> <p>Known/tested bad values:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2019-09-04T21:15:13.438 xcu_b2 w ionpump2errors=0x8000,"Suspect live channel" 2019-09-04T21:18:25.529 xcu_b2 w ionpump2errors=0x0020,"Interlock cable" </pre> </div></div> <p>vs.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2019-09-04T21:18:34.026 xcu_b2 i ionpump2errors=0x0000,OK </pre> </div></div> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a> mentioned during the ICS+DRP/SPS telecon 2019-09-18 that this ticket is most likely already implemented. He will confirm (and close if appropriate) in the next day or so.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=arnaud.lefur" class="user-hover" rel="arnaud.lefur">arnaud.lefur</a> to review and close.</p> <p><span class="image-wrap" style=""><img src="https://pfspipe.ipmu.jp/jira/secure/attachment/12517/12517_STS_ionPump1_errorString.png" style="border: 0px solid black" /></span></p> <p>Implemented and tracked in STS as shown above from last cooldown.</p> Relates INSTRM-758 INSTRM-773 Development Rank 0|02qpuc:i6001gtu Sprint SM1-2019 N Story Points 1.0