[INSTRM-1019] Failed to get a visit in 10s Created: 19/Jun/20 Updated: 05/Aug/20 Resolved: 01/Jul/20 |
|
| Status: | Done |
| Project: | Instrument control development |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Blocker |
| Reporter: | fmadec | Assignee: | fmadec |
| Resolution: | Done | Votes: | 0 |
| Labels: | SM1, SPS | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Sprint: | SM1PD-2020 G | ||||||||
| Description |
|
Hexapod sequence stop because of the error: failed to get a visit in 10s
IIC or sequencePanel seems blocked, the switch off of the HgAr lamp has not been done one sequence is in light blue but nothing happen |
| Comments |
| Comment by fmadec [ 22/Jun/20 ] |
|
it happened again on Friday night
2020-06-19 11:46:23 iic f text="command failed: RuntimeError('Sub-command has failed.. sequence aborted..',) in processSubCmd() at /software/mhs/products/Linux64/ics_iicActor/1.1.2/python/ics/iicActor/utils/sequencing.py:263"
so HgAr lamp remained ON for nothing during all the week-end...
|
| Comment by cloomis [ 25/Jun/20 ] |
|
The reqframes RPC call by the g2cam library did take too long a few times, confirmed by the timestamps on the Subaru side. The Gen2 simulator we have been using runs on a shared machine in Hilo; they are installing a PFS-only Gen2 simulator on a summit machine. We will need to see what effect that has on performance. FWIW, here are all instances of reqframes delays over 6s since 2019-12-16: 2020-06-19 06:13:43.504 6.988 2020-06-11 10:39:04.736 8.48 2020-06-24 09:16:51,372 13.815 2020-06-19 08:15:40.965 18.792 2020-06-19 08:55:19.056 20.096 2020-06-19 11:45:59,894 38.717 2020-06-19 08:53:04,728 154.415 And for the record, with the current server the minimum time to allocate a new set of frames is about 1.25s |
| Comment by cloomis [ 01/Jul/20 ] |
|
The Gen2 simulator for PFS was changed to a dedicated summit machine. After that we took 300 frames. The minimum time was 0.25s (vs. 1.25s), and the outliers above 0.4s were: 2020-06-26 12:57:21,592 1 0.413 2020-06-29 18:21:27,974 1 0.445 2020-06-29 14:52:27,392 1 0.456 2020-06-29 19:54:59,409 1 0.473 2020-06-29 19:19:26,529 1 0.482 2020-06-29 18:53:15,866 1 0.496 2020-06-29 20:28:40,353 1 0.512 2020-06-26 12:49:27,097 1 0.599 2020-06-29 13:23:39,601 1 0.675 2020-06-29 17:45:56,221 1 0.7 vs. the list above. |