https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/INSTRM-918 Instrument control development <p> </p> <p><font color="#000000">03:53:07.430948 spsait w text="enu_sm1 shutters failed: FysomError('event init inappropriate in current state FAILED',) in fn() at /software/conda/lib/python3.6/site-packages/fysom/_<em>init</em>_.py:272"</font></p> <p><font color="#000000">Don't know why they failed</font></p> <p>Basically biasha board is stuck in a peculiar state where it doesn't answer correctly.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>alefur@shell-ics:~$ oneCmd.py enu_sm1 biasha raw=status 2020-08-13T16:00:44.454 sent enu_sm1 biasha raw=status (from alefur_shell_ics) 2020-08-13T16:00:44.478 enu_sm1 : text="returned: 30" alefur@shell-ics:~$ oneCmd.py enu_sm1 biasha raw=statword 2020-08-13T16:01:01.772 sent enu_sm1 biasha raw=statword (from alefur_shell_ics) 2020-08-13T16:01:01.792 enu_sm1 : text="returned: n" </pre> </div></div> <p> The only fix is to power cycle the board</p> INSTRM-918

SM1 at Subaru : shutters failed

Bug Normal Done Done arnaud.lefur fmadec SM1 SPS Wed, 11 Mar 2020 14:03:03 +0000 Mon, 21 Jun 2021 18:24:02 +0000 Mon, 21 Jun 2021 18:24:02 +0000 ics_enuActor 0 4 <p>just did disconnect/connect , status became ok.</p> <p> </p> <p>a dark test image works.</p> <p>but then I started a sequence but it failed again:</p> <p> </p> <p><font color="#000000"> 04:06:44.656414 spsait w text="enu_sm1 shutters failed: RuntimeError('biasha has replied nok, red_open inappropriate in current state ',) in _gotoState() at /software/mhs/products/Linux64/ics_enuActor/1.0.0/python/enuActor/Controllers/biasha.py:344" 04:06:44.656966 spsait w text="command failed: RuntimeError('Exposure has failed',) in expose() at /software/mhs/devel/ics_spsActor/python/spsActor/Controllers/expose.py:58" 04:06:44.711224 spsait f text="command failed: RuntimeError('subCmd has failed.. sequence aborted..',) in processSubCmd() at /software/mhs/products/Linux64/ics_spsaitActor/1.0.3/python/spsaitActor/utils/experiment.py:80"</font></p> <p> </p> <p> </p> <p>It's seems that the arduino is stuck in a weird state, some command are not recognized correctly. </p> <p>Power cycle fixed it, but that's the second time it's happening at Subaru and we never seen it at LAM, a bit worrysome...</p> <p>it happened again today</p> <p>07:07:55.644614 spsait w text="enu_sm1 shutters failed: RuntimeError('biasha has replied nok, shut_open inappropriate in current state ',) in _gotoState() at /software/mhs/products/Linux64/ics_enuActor/1.0.0/python/enuActor/Controllers/biasha.py:344"<br/> 07:07:55.645212 spsait w text="command failed: RuntimeError('Exposure has failed',) in expose() at /software/mhs/devel/ics_spsActor/python/spsActor/Controllers/expose.py:58"<br/> 07:07:55.699574 spsait f text="command failed: RuntimeError('subCmd has failed.. sequence aborted..',) in processSubCmd() at /software/mhs/products/Linux64/ics_spsaitActor/1.0.3/python/spsaitActor/utils/experiment.py:80"</p> <p>As we heard during today's ICS/SpS telecon: not much can be done without significant software changes. No longer required for the spsait-&gt;Subaru migration. Problem appears on an exposure-by-exposure basis, so the worst that can happen is that one exposure is lost. Nominal situation then recovered after a power-cycle.</p> <p>Will lower priority for this.</p> <p>From <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=arnaud.lefur" class="user-hover" rel="arnaud.lefur">arnaud.lefur</a>: not much can be done about this. We just need to monitor the situation regularly.</p> <p>the bug again using only blue shutters.<br/> board was stuck and needed to be power-cycled.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>alefur@shell-ics:~$ oneCmd.py enu_sm1 biasha raw=status 2020-08-13T16:00:44.454 sent enu_sm1 biasha raw=status (from alefur_shell_ics) 2020-08-13T16:00:44.478 enu_sm1 : text="returned: 30" alefur@shell-ics:~$ oneCmd.py enu_sm1 biasha raw=statword 2020-08-13T16:01:01.772 sent enu_sm1 biasha raw=statword (from alefur_shell_ics) 2020-08-13T16:01:01.792 enu_sm1 : text="returned: n" </pre> </div></div> <p>happened again today !</p> <p>Requires a firmware update and discussions with LAM support on this.</p> <p>Arnaud will organize a meeting with LAM support (Philip?) to address this.</p> <p>And on 2021-01-16:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2021-01-16 22:37:05.780Z biasha 10 bufferedSocket.py:64 sending b'init\r\n' 2021-01-16 22:37:05.795Z biasha 10 bufferedSocket.py:133 received 'ok' 2021-01-16 22:37:05.798Z biasha 10 bufferedSocket.py:64 sending b'blue_open\r\n' 2021-01-16 22:37:05.811Z biasha 10 bufferedSocket.py:133 received 'nok' </pre> </div></div> <p>A similar problem is occurring at LAM (SM2). Arnaud will investigate this further at LAM.</p> <p>Arnaud starting testing at LAM. Ongoing.</p> <p>currently running a fixed version on SM3.</p> <p>Problem not seen so far at LAM.</p> <p>Problem after firmware update at LAM, no further problems have been seen.</p> <p>Following discussions with Fabrice, Arnaud and Craig: will update the firmware further and coordinate with Subaru in ~ 1 month to install the updated firmware.</p> <p>Problem occured this morning at LAM with the new firmware version:</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2021-06-02 07:15:05.729Z cmds             20 CommandLink.py:121 &gt; 2 1048 f text="command failed: RuntimeError('error : shutter switch timeoutn') in _gotoState() at /software/mhs/p roducts/Linux64/ics_enuActor/1.1.8/python/enuActor/Controllers/biasha.py:347" 2021-06-02 07:15:53.886Z cmds             20 CommandLink.py:121 &gt; 2 1050 f text="command failed: FysomError('event init inappropriate in current state FAILED') in fn() at /softwar e/conda/envs/conda-ics/lib/python3.7/site-packages/fysom/__init__.py:272" 2021-06-02 07:16:08.940Z cmds             20 CommandLink.py:121 &gt; 0 0 w text="command failed: RuntimeError('statword 010100  does not match current state') in shutterStatus() at / software/mhs/products/Linux64/ics_enuActor/1.1.8/python/enuActor/Controllers/biasha.py:237" 2021-06-02 07:16:23.993Z cmds             20 CommandLink.py:121 &gt; 0 0 w text="command failed: RuntimeError('statword 010100  does not match current state') in shutterStatus() at / software/mhs/products/Linux64/ics_enuActor/1.1.8/python/enuActor/Controllers/biasha.py:237" 2021-06-02 07:16:37.191Z cmds             20 CommandLink.py:121 &gt; 2 1052 f text="command failed: FysomError('event init inappropriate in current state FAILED') in fn() at /softwar e/conda/envs/conda-ics/lib/python3.7/site-packages/fysom/__init__.py:272" 2021-06-02 07:16:52.244Z cmds             20 CommandLink.py:121 &gt; 0 0 w text="command failed: RuntimeError('statword 010100  does not match current state') in shutterStatus() at / software/mhs/products/Linux64/ics_enuActor/1.1.8/python/enuActor/Controllers/biasha.py:237" 2021-06-02 07:17:07.297Z cmds             20 CommandLink.py:121 &gt; 0 0 w text="command failed: RuntimeError('statword 010100  does not match current state') in shutterStatus() at / software/mhs/products/Linux64/ics_enuActor/1.1.8/python/enuActor/Controllers/biasha.py:237" 2021-06-02 07:17:20.296Z cmds             20 CommandLink.py:121 &gt; 2 1054 f text="command failed: FysomError('event init inappropriate in current state FAILED') in fn() at /softwar e/conda/envs/conda-ics/lib/python3.7/site-packages/fysom/__init__.py:272" 2021-06-02 07:17:35.349Z cmds             20 CommandLink.py:121 &gt; 0 0 w text="command failed: RuntimeError('statword 010100  does not match current state') in shutterStatus() at / so</pre> </div></div> <p> </p> <p>It looks like another issue.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2021-06-02 07:15:00.126Z cmds 20 CommandLink.py:121 &gt; 2 1048 i elapsedTime=6.43 2021-06-02 07:15:02.127Z cmds 20 CommandLink.py:121 &gt; 2 1048 i elapsedTime=8.43 2021-06-02 07:15:03.701Z cmds 20 CommandLink.py:121 &gt; 0 0 i biashaFSM=ONLINE,BUSY 2021-06-02 07:15:03.701Z cmds 20 CommandLink.py:121 &gt; 0 0 i metaFSM=ONLINE,BUSY 2021-06-02 07:15:03.701Z biasha 10 bufferedSocket.py:64 sending b'red_close\r\n' 2021-06-02 07:15:05.722Z biasha 10 bufferedSocket.py:133 received 'shutter switch timeoutnok' 2021-06-02 07:15:05.722Z cmds 20 CommandLink.py:121 &gt; 0 0 i biashaFSM=ONLINE,FAILED 2021-06-02 07:15:05.722Z cmds 20 CommandLink.py:121 &gt; 0 0 i metaFSM=ONLINE,FAILED </pre> </div></div> <p>An exposure was being taken, the red shutter did not close as it's supposed to.<br/> But the board itself was responding well:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2021-06-02 07:16:08.936Z biasha 10 bufferedSocket.py:133 received '84ok' 2021-06-02 07:16:08.936Z cmds 20 CommandLink.py:121 &gt; 0 0 i shb=0,1,0 2021-06-02 07:16:08.937Z cmds 20 CommandLink.py:121 &gt; 0 0 i shr=1,0,0 2021-06-02 07:16:08.937Z cmds 20 CommandLink.py:121 &gt; 0 0 w shutters=undef 2021-06-02 07:16:08.940Z cmds 20 CommandLink.py:121 &gt; 0 0 w text="command failed: RuntimeError('statword 010100 does not match current state') in shutterStatus() at /software/mhs/products/Linux64/ics_enuActor/1.1.8/python/enuActor/Controllers/biasha.py:237" 2021-06-02 07:16:23.941Z cmds 20 CommandLink.py:121 &gt; 0 0 i biashaFSM=ONLINE,FAILED 2021-06-02 07:16:23.941Z cmds 20 CommandLink.py:121 &gt; 0 0 i biashaMode=operation 2021-06-02 07:16:23.942Z biasha 10 bufferedSocket.py:64 sending b'status\r\n' 2021-06-02 07:16:23.947Z biasha 10 bufferedSocket.py:133 received '0ok' 2021-06-02 07:16:23.947Z cmds 20 CommandLink.py:121 &gt; 0 0 i biasha=0 </pre> </div></div> <p>This is reflected here, in biasha=0 both shutters are supposed to be closed, but red was actually still open.</p> <p>It would have been interesting to see whether a <em>biasha init</em> command could fix the issue.</p> <p>after 3 month of test, this mysterious bug never came back !<br/> So let's declare victory now and close this ticket.<br/> Will open a new one if it appears to be necessary.</p> Relates INSTRM-1239 INSTRM-1226 INSTRM-1232 Development Rank 0|zzs4ye:i Sprint SM1PD-2020 C SM1PD-2020 D SM1PD-2020 G SM1PD-2020 G2 SM1PD-2020 H SM1PD-2020 J SM1PD-2021 A 3 SM1PD-2021 A 4 SM1PD-2021 A 5 SM1PD-2021 A 6 SM1PD-2021 A 11 Story Points 2.0