Uploaded image for project: 'Instrument control development'
  1. Instrument control development
  2. INSTRM-2573

When agActor fails errors aren't being propagated and guiding goes into failure mode

    XMLWordPrintable

    Details

    • Sprint:
      preRun24Sep2025

      Description

      From yuki.moritani

       

      Last night ics_agccActor failed to read/save image for visit=126418, agc_exposure_id=725567

       less /software/mhs/logs/agcc/stdio-2025-05-25T00\:55\:46.log
      
      2025-05-27 15:14:28.450Z agcc             20 dbRoutinesAGCC.py:110 Table is prepared.
      2025-05-27 15:14:28.471Z agcc             20 dbRoutinesAGCC.py:145 Bulk insert completed successfully.
      2025-05-27 15:14:28.472Z cmds             20 CommandLink.py:122 > 2 14190 i agc_exposing=0
      2025-05-27 15:14:28.472Z cmds             20 CommandLink.py:122 > 2 14190 i agc_frameid=725566
      2025-05-27 15:14:28.793Z cmds             20 CommandLink.py:122 > 2 14190 i agc_fitsfile="/data/raw/2025-05-27/agcc/agcc_126418_00725566.fits",1748358864.5
      2025-05-27 15:14:28.793Z cmds             20 CommandLink.py:122 > 2 14190 i text="AG images are NOT written into /data/raw/2025-05-27/agcc/agcc_126418_00725566.fits"
      2025-05-27 15:14:28.794Z cmds             20 CommandLink.py:122 > 2 14190 :
      2025-05-27 15:14:29.861Z cmds             20 Actor.py:525 new cmd: expose object exptime=3.0 centroid=1 visit=126418 threadDelay=100
      .......
      2025-05-27 15:14:33.980Z agcc            20 dbRoutinesAGCC.py:110 Table is prepared.
      2025-05-27 15:14:34.000Z agcc            20 dbRoutinesAGCC.py:145 Bulk insert completed successfully.
      2025-05-27 15:14:48.514Z cmds            20 Actor.py:525 new cmd: expose object exptime=3.0 centroid=1 visit=126418 threadDelay=100 

      For previous one, after "Bulk insert complete successfully", it saved the file. However, 725567, agccActor stacked.Indeed agsActor got timeout error.

      2025-05-27 15:14:48.497Z actor           20 main.py:101 reply=ag.ag 191 ag F timeout;actor="agcc";cmd="expose object exptime=3.0 centroid=1 visit=126418 threadDelay=100"
      2025-05-27 15:14:48.497Z actor           20 main.py:102 didFail=True
      2025-05-27 15:14:48.498Z ag              40 ag.py:437 AgThread.run:
      Traceback (most recent call last):
       File "/software/mhs/products/Linux64/ics_agActor/1.1.26/python/agActor/Controllers/ag.py", line 324, in run
         result.get()
       File "/software/mhs/products/Linux64/ics_agActor/1.1.26/python/agActor/main.py", line 104, in get
         raise Exception('command failed: params={}'.format(self.params))
      Exception: command failed: params={'actor': 'agcc', 'cmdStr': 'expose object exptime=3.0 centroid=1 visit=126418 threadDelay=100', 'timeLim': 18, 'kwargs': {}}
      2025-05-27 15:14:48.509Z ag              20 ag.py:276 AgThread.run: Setting design for ON/ONCE/REF_SKY
      2025-05-27 15:14:48.509Z cmds            20 CommandLink.py:122 > 0 0 i exposureTime=3000 

      However, agActor continued to send next exposure, never succeeded. (So we didn't guide from this image..)
      We didn't get error from agActor... could you sort out those issue?
      At least agActor should return failure... so that we recognized the problem.

        Attachments

          Activity

            People

            • Assignee:
              wtgee Wilfred Gee
              Reporter:
              wtgee Wilfred Gee
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: