-
Type:
Task
-
Status: In Progress (View Workflow)
-
Priority:
Normal
-
Resolution: Unresolved
-
Component/s: ics_agActor, ics_agccActor
-
Labels:
-
Sprint:preRun24Sep2025
From yuki.moritani
Last night ics_agccActor failed to read/save image for visit=126418, agc_exposure_id=725567
less /software/mhs/logs/agcc/stdio-2025-05-25T00\:55\:46.log 2025-05-27 15:14:28.450Z agcc 20 dbRoutinesAGCC.py:110 Table is prepared. 2025-05-27 15:14:28.471Z agcc 20 dbRoutinesAGCC.py:145 Bulk insert completed successfully. 2025-05-27 15:14:28.472Z cmds 20 CommandLink.py:122 > 2 14190 i agc_exposing=0 2025-05-27 15:14:28.472Z cmds 20 CommandLink.py:122 > 2 14190 i agc_frameid=725566 2025-05-27 15:14:28.793Z cmds 20 CommandLink.py:122 > 2 14190 i agc_fitsfile="/data/raw/2025-05-27/agcc/agcc_126418_00725566.fits",1748358864.5 2025-05-27 15:14:28.793Z cmds 20 CommandLink.py:122 > 2 14190 i text="AG images are NOT written into /data/raw/2025-05-27/agcc/agcc_126418_00725566.fits" 2025-05-27 15:14:28.794Z cmds 20 CommandLink.py:122 > 2 14190 : 2025-05-27 15:14:29.861Z cmds 20 Actor.py:525 new cmd: expose object exptime=3.0 centroid=1 visit=126418 threadDelay=100 ....... 2025-05-27 15:14:33.980Z agcc 20 dbRoutinesAGCC.py:110 Table is prepared. 2025-05-27 15:14:34.000Z agcc 20 dbRoutinesAGCC.py:145 Bulk insert completed successfully. 2025-05-27 15:14:48.514Z cmds 20 Actor.py:525 new cmd: expose object exptime=3.0 centroid=1 visit=126418 threadDelay=100For previous one, after "Bulk insert complete successfully", it saved the file. However, 725567, agccActor stacked.Indeed agsActor got timeout error.
2025-05-27 15:14:48.497Z actor 20 main.py:101 reply=ag.ag 191 ag F timeout;actor="agcc";cmd="expose object exptime=3.0 centroid=1 visit=126418 threadDelay=100" 2025-05-27 15:14:48.497Z actor 20 main.py:102 didFail=True 2025-05-27 15:14:48.498Z ag 40 ag.py:437 AgThread.run: Traceback (most recent call last): File "/software/mhs/products/Linux64/ics_agActor/1.1.26/python/agActor/Controllers/ag.py", line 324, in run result.get() File "/software/mhs/products/Linux64/ics_agActor/1.1.26/python/agActor/main.py", line 104, in get raise Exception('command failed: params={}'.format(self.params)) Exception: command failed: params={'actor': 'agcc', 'cmdStr': 'expose object exptime=3.0 centroid=1 visit=126418 threadDelay=100', 'timeLim': 18, 'kwargs': {}} 2025-05-27 15:14:48.509Z ag 20 ag.py:276 AgThread.run: Setting design for ON/ONCE/REF_SKY 2025-05-27 15:14:48.509Z cmds 20 CommandLink.py:122 > 0 0 i exposureTime=3000However, agActor continued to send next exposure, never succeeded. (So we didn't guide from this image..)
We didn't get error from agActor... could you sort out those issue?
At least agActor should return failure... so that we recognized the problem.