[INSTRM-83] Echo mismatch from CCD temperature monitoring Created: 01/Feb/17  Updated: 13/Jul/18  Resolved: 13/Jul/18

Status: Done
Project: Instrument control development
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major
Reporter: arnaud.lefur Assignee: cloomis
Resolution: Done Votes: 0
Labels: SM1
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File bug_ccd2.png     PNG File bug_ccd.png    
Issue Links:
Duplicate
is duplicated by INSTRM-36 SerialWriteTimeout on FEE Won't Fix
Relates
relates to INSTRM-84 FEE unresponsive, even after power-cy... Won't Fix
relates to INSTRM-190 CCD read command does not finish prop... Won't Fix
relates to INSTRM-226 Checking ccd temperature before start... Won't Fix
Epic Link: SM1 Phase One

 Description   

After a while, a error occurs on the ccd status command and cause the temperature monitoring to stop.
It's seem that the buffer isn't synchronised anymore.

The only solution for now is to reboot the FEE.

17:28:57.757250 .hub 0 cmds d CmdQueued=180320,1485883737.75,"client.alefur",2,"ccd_r1",5,"status"
17:28:57.777622 client.alefur 2 ccd_r1 w text='pathetic version string: unknown'
17:28:57.777876 client.alefur 2 ccd_r1 i version="unknown"
17:28:57.778100 client.alefur 2 ccd_r1 i text="exposure=None"
17:28:57.778327 client.alefur 2 ccd_r1 f text="command failed: RuntimeError(\"command echo mismatch. sent :'rt,FEE': rcvd :'264.87':\",) at /home/hostpfs/cpl/git/2016-11-30/ics_xcu_fpga/python/xcu_fpga/fee/feeControl.py:788"
17:28:57.778562 .hub 0 cmds d CmdDone=180320,"f

 Comments   
Comment by arnaud.lefur [ 02/Feb/17 ]

Fee is doing okay so far, but this is worrysome.

Comment by arnaud.lefur [ 17/Feb/17 ]

It happens two times yesterday, and again during the night.
To fix it I did a power cycle and then a disconnect/connect controller for the first two times.
But notice, that for the last time, I did only a connect/disconnect and it was working also.
This means that the power cycling may not be required, and that it could be handled by software.

Comment by arnaud.lefur [ 17/Feb/17 ]

Forget about that, I was using the wrong command.
It's ccd_r1 monitor controllers=temps period=0

Another bug which could be related or not.
The monitoring isn't stopping when I ask for it.
(I did a fee disconnect that's why you have this KeyError exception).

09:52:50.199016 .hub 0 cmds d CmdIn="client.alefur","ccd_r1","monitor controllers=fee period=0"
09:52:50.199732 .hub 0 cmds d CmdQueued=1397466,1487325170.19,"client.alefur",26,"ccd_r1",95,"monitor controllers=fee period=0"
09:52:50.221279 client.alefur 26 ccd_r1 w text="adjusted fee loop to 0s"
09:52:50.221893 client.alefur 26 ccd_r1 :
09:52:50.222425 .hub 0 cmds d CmdDone=1397466,":"
09:52:59.733435 .ccd_r1 6187 ccd_r1 f text="command failed: KeyError('fee',) at python/ccdActor/main.py:108"
09:53:14.707476 .ccd_r1 6188 ccd_r1 f text="command failed: KeyError('fee',) at python/ccdActor/main.py:108"
09:53:29.721332 .ccd_r1 6189 ccd_r1 f text="command failed: KeyError('fee',) at python/ccdActor/main.py:108"

Comment by arnaud.lefur [ 31/Mar/17 ]

It happened again this morning.
A simple disconnect / connect of the fee controller was enough to fix it.

Comment by arnaud.lefur [ 04/May/17 ]


Instead of commenting each time this bug happens I have analysed ccd logfile since we have put the science detector in late november.
Here is a plot which show the bug frequency since then.

Comment by arnaud.lefur [ 04/May/17 ]


Note that there is 3 kind of bugs which occur more or less in same timing :

  • "command failed: RuntimeError(\"command echo mismatch.",
  • "command failed: ValueError('could not convert string to float",
  • "command failed: SerialTimeoutException('Write timeout',)"
Comment by cloomis [ 13/Jul/18 ]

The exposure runs in a separate thread from the command handler. So when the exposure timer ends and the device is read out two threads are communicating with the device. Added a lock around a complete command and response to mediate.

Merged in 6785776, tag 70.4.4

Generated at Sat Feb 10 16:21:23 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.