[#INSTRM-1978] Check when telescope status in agc

[INSTRM-1978] Check when telescope status in agc_exposure table is obtained Created: 12/May/23 Updated: 19/Jul/23 Resolved: 19/Jul/23
Status:	Done
Project:	Instrument control development
Component/s:	ics_agccActor
Affects Version/s:	None
Fix Version/s:	None

Type:

Task

Priority:

Normal

Reporter:

yuki.moritani

Assignee:

chyan

Resolution:

Done

Votes:

Labels:

EngRun

Remaining Estimate:

Not Specified

Time Spent:

Not Specified

Original Estimate:

Not Specified

Sprint:

Eng12July

Description

Telescope statuses stored in agc_exposure table look different from what is expected for the time when the image is taken.
The below is a copy from message on Slack, pfi-ics channel:

****
I have a question about the telescope status stored in {{agc_exposure }}table. We look for the best-focus position using AG; taking AG images at different hexapod z-position and getting the position where the spot size is the same between w/ and w/o glasses. Looking the data in opDB, it seems that telescope status is different from expected.
The below table is a example: we took three frames at each hexapod-z position, -0.3, -0.2, -0.1, 0, +0.1, +0.2, +0.3. The spot size (the four columns on the right are average of spot size stored in agc_data table) seems to follow telescope movement; the size changes every three frames. On the other hand, m2pos3 (hexapod position) looks shifted by one... I guess the telescope status is for the last exposure.
Could you tell when you read telescope status, and write on opDB?

agc_exposre_id m2pos3 l_x l_y r_x r_y 
 191084 5.09 2.09 1.83 3.91 4.52
 191085 4.79 2.06 1.73 3.72 4.09
 191086 4.79 2.32 2.12 4.05 5.58
 191087 4.79 1.65 1.51 2.73 3.58
 191088 4.89 1.76 1.51 2.86 3.52
 191089 4.89 1.69 1.47 2.81 3.40
 191090 4.89 1.97 1.59 2.21 2.03
 191091 4.99 1.80 1.49 2.07 2.19
 191092 4.99 1.77 1.49 2.20 2.03
 191093 4.99 2.03 1.88 1.78 1.77
 191094 5.09 1.58 1.42 1.40 1.38
 191095 5.09 2.31 1.94 1.89 1.77
 191096 5.09 2.44 2.08 1.98 1.88
 191097 5.19 2.27 1.91 1.67 1.52
 191098 5.19 2.65 2.31 1.96 1.68
 191099 5.19 3.79 4.41 2.38 2.05
 191100 5.29 3.68 4.04 2.25 2.12
 191101 5.29 3.75 4.27 2.15 2.06
 191102 5.29 4.16 4.70 2.35 2.18
 191103 5.39 4.82 5.34 2.77 2.22
 191104 5.39 4.40 4.06 2.31 1.83

Comments

Comment by chyan [ 18/May/23 ]

Looking at the code, the agccActor writes agc_exposure table after finishing every exposure. However, the pfs_visit_id and exposure time are not correctly inserted. If you check the agc_expose_id = 19084, you will see the pfs_visit_id is 0. In this case, the telescope information will be wrong because there is no pfs_visit_id related to this agc_expose_id.

I have populated a fix. Yoshida, Hiroshige please make sure that not using visit=0 when calling agccActor.

Comment by yuki.moritani [ 19/May/23 ]

Thank you for investigating. Have you checked when the information which agccActor writes are obtained? I think visit=0 option is necessary to test AG camera by running long-time as we usually do to test E-boxes.

Comment by yuki.moritani [ 20/May/23 ]

Just in case... here is a copy from agcc Actor and gen2 Actor log for agc_expose_id = 191084. It seems that agccActor sends query slightly earlier than gen2 gets the status, given that both have the same clock:

/software/mhs/logs/agcc/stdio-2023-04-30T05\:27\:55.log

2023-04-29 19:34:24.417Z cmds 20 CommandLink.py:121 > 2 6 i text="read magFit = [-2.227, 27.66]"
2023-04-29 19:34:24.417Z cmds 20 CommandLink.py:121 > 2 6 i text="found cameras: [0, 1, 2, 3, 4, 5]"
2023-04-29 19:34:24.417Z cmds 20 CommandLink.py:121 > 2 6 i text="Number of AG cameras = 6"
2023-04-29 19:34:24.418Z cmds 20 CommandLink.py:121 > 2 6 i text="[1] MicroLine ML4720 SN=ML0424912 status=READY temp= 3.6"
2023-04-29 19:34:24.418Z cmds 20 CommandLink.py:121 > 2 6 i text="[2] MicroLine ML4720 SN=ML0374912 status=READY temp= 3.2"
2023-04-29 19:34:24.419Z cmds 20 CommandLink.py:121 > 2 6 i text="[3] MicroLine ML4720 SN=ML0384912 status=READY temp= 3.4"
2023-04-29 19:34:24.420Z cmds 20 CommandLink.py:121 > 2 6 i text="[4] MicroLine ML4720 SN=ML0394912 status=READY temp= 1.8"
2023-04-29 19:34:24.420Z cmds 20 CommandLink.py:121 > 2 6 i text="[5] MicroLine ML4720 SN=ML0404912 status=READY temp= 1.5"
2023-04-29 19:34:24.421Z cmds 20 CommandLink.py:121 > 2 6 i text="[6] MicroLine ML4720 SN=ML0414912 status=READY temp= 3.7"
2023-04-29 19:34:24.421Z cmds 20 CommandLink.py:121 > 2 6 i text="Receive expose command"
2023-04-29 19:34:24.431Z opdb 20 opdb.py:440 fetching sql: COPY (select agc_exposure_id from agc_exposure ORDER BY agc_exposure_id DESC LIMIT 1) TO STDOUT WITH (FORMAT csv, HEADER)
2023-04-29 19:34:24.440Z cmds 20 CommandLink.py:121 > 2 6 i text="Getting agc_exposure_id = 191084 from opDB"
2023-04-29 19:34:24.440Z cmds 20 CommandLink.py:121 > 2 6 i text="Recording agc_exposure_id = 191084 to /software/mhs/data/agcc/nframe.txt"
2023-04-29 19:34:24.449Z opdb 20 opdb.py:440 fetching sql: COPY (select pfs_visit_id, altitude, azimuth, insrot, adc_pa, m2_pos3 FROM tel_status ORDER BY pfs_visit_id DESC limit 1) TO STDOUT WITH (FORMAT csv, HEADER)
2023-04-29 19:34:24.461Z opdb 20 opdb.py:440 fetching sql: COPY (select pfs_visit_id, outside_temperature, outside_pressure, outside_humidity FROM env_condition ORDER BY pfs_visit_id DESC limit 1) TO STDOUT WITH (FORMAT csv, HEADER)
2023-04-29 19:34:24.479Z cmds 20 CommandLink.py:121 > 2 6 i agc_exposing=6
2023-04-29 19:34:24.480Z cmds 20 CommandLink.py:121 > 2 6 i agc1_stat=BUSY
2023-04-29 19:34:24.480Z cmds 20 CommandLink.py:121 > 2 6 i agc2_stat=BUSY
2023-04-29 19:34:24.480Z cmds 20 CommandLink.py:121 > 2 6 i agc3_stat=BUSY
2023-04-29 19:34:24.481Z cmds 20 CommandLink.py:121 > 2 6 i agc4_stat=BUSY
2023-04-29 19:34:24.481Z cmds 20 CommandLink.py:121 > 2 6 i agc5_stat=BUSY
2023-04-29 19:34:24.481Z cmds 20 CommandLink.py:121 > 2 6 i agc6_stat=BUSY
2023-04-29 19:34:27.833Z cmds 20 CommandLink.py:121 > 2 6 i text="AGC[3]: Retrieve camera data in 3.35s"
2023-04-29 19:34:27.833Z cmds 20 CommandLink.py:121 > 2 6 i text="AGC[2]: Retrieve camera data in 3.35s"
2023-04-29 19:34:27.834Z cmds 20 CommandLink.py:121 > 2 6 i agc3_stat=READY
2023-04-29 19:34:27.834Z cmds 20 CommandLink.py:121 > 2 6 i text="AGC[6]: Retrieve camera data in 3.35s"
2023-04-29 19:34:27.839Z cmds 20 CommandLink.py:121 > 2 6 i text="AGC[4]: Retrieve camera data in 3.36s"
2023-04-29 19:34:27.840Z cmds 20 CommandLink.py:121 > 2 6 i text="AGC[5]: Retrieve camera data in 3.36s"
2023-04-29 19:34:27.840Z cmds 20 CommandLink.py:121 > 2 6 i agc2_stat=READY
2023-04-29 19:34:27.847Z cmds 20 CommandLink.py:121 > 2 6 i agc6_stat=READY
2023-04-29 19:34:27.852Z cmds 20 CommandLink.py:121 > 2 6 i agc5_stat=READY
2023-04-29 19:34:27.852Z cmds 20 CommandLink.py:121 > 2 6 i agc4_stat=READY
2023-04-29 19:34:27.864Z cmds 20 CommandLink.py:121 > 2 6 i text="AGC[1]: Retrieve camera data in 3.38s"
2023-04-29 19:34:27.864Z cmds 20 CommandLink.py:121 > 2 6 i agc1_stat=READY
2023-04-29 19:34:28.672Z cmds 20 CommandLink.py:121 > 2 6 i text="AGC[3]: find 15 objects"
2023-04-29 19:34:28.672Z cmds 20 CommandLink.py:121 > 2 6 i text="wrote centroids to database"

/software/mhs/logs/actors/gen2/2023-04-30T02\:56\:31.011.log

2023-04-30 05:34:24.396Z cmdr 20 CmdrConnection.py:218 queueing command ag(acquire_field otf visit_id=0 exposure_time=3000)
2023-04-30 05:34:25.904Z cmds 20 Actor.py:525 new cmd: updateTelStatus caller=ag
2023-04-30 05:34:25.930Z cmds 20 CommandLink.py:121 > 2 29069 d text="updating opdb.tel_status with visit=92671, sequence=2"
2023-04-30 05:34:25.946Z cmds 20 CommandLink.py:121 > 2 29069 i inst_ids="NAOJ","Subaru","PFS"
2023-04-30 05:34:25.946Z cmds 20 CommandLink.py:121 > 2 29069 i program="o23016","SPEC_ENG","Observation","\"[S] Tamura, Moritani, Kawanomoto, Yoshida, Le Fur, Lupton\""
2023-04-30 05:34:25.946Z cmds 20 CommandLink.py:121 > 2 29069 i object="ras08-10","08:00:00.001","-09:59:59.970","08:00:00.001","-09:59:59.970"
2023-04-30 05:34:25.946Z cmds 20 CommandLink.py:121 > 2 29069 i pointing="08:00:00.000","-10:00:00.000"
2023-04-30 05:34:25.946Z cmds 20 CommandLink.py:121 > 2 29069 i offsets=0.0000,0.0000
2023-04-30 05:34:25.946Z cmds 20 CommandLink.py:121 > 2 29069 i telDither=0.000,0.000,0.000
2023-04-30 05:34:25.947Z cmds 20 CommandLink.py:121 > 2 29069 i telGuide=0.000,0.000,0.000
2023-04-30 05:34:25.947Z cmds 20 CommandLink.py:121 > 2 29069 i coordinate_system_ids="FK5",180.0,2000.0
2023-04-30 05:34:25.947Z cmds 20 CommandLink.py:121 > 2 29069 i tel_axes=222.6101,50.7956,39.20442772,1.290
2023-04-30 05:34:25.947Z cmds 20 CommandLink.py:121 > 2 29069 i tel_rot=0.0,40.306659
2023-04-30 05:34:25.947Z cmds 20 CommandLink.py:121 > 2 29069 i tel_focus="P_OPT2","PRIME",2.9
2023-04-30 05:34:25.947Z cmds 20 CommandLink.py:121 > 2 29069 i tel_adc="IN",9.28
2023-04-30 05:34:25.947Z cmds 20 CommandLink.py:121 > 2 29069 i dome_env=27.700,621.000,274.050,0.100
2023-04-30 05:34:25.947Z cmds 20 CommandLink.py:121 > 2 29069 i outside_env=29.800,621.000,272.750,3.800
2023-04-30 05:34:25.947Z cmds 20 CommandLink.py:121 > 2 29069 i m2="Opt",-2.810068,-4.367825,4.788862
2023-04-30 05:34:25.947Z cmds 20 CommandLink.py:121 > 2 29069 i m2rot=1.2000,2.5000,0.0000
2023-04-30 05:34:25.947Z cmds 20 CommandLink.py:121 > 2 29069 i pfuOffset=-1.600,-2.500,2.900
2023-04-30 05:34:25.947Z cmds 20 CommandLink.py:121 > 2 29069 i autoguider="OFF"
2023-04-30 05:34:25.947Z cmds 20 CommandLink.py:121 > 2 29069 i conditions="Fine",9998.000,1.000
2023-04-30 05:34:25.947Z cmds 20 CommandLink.py:121 > 2 29069 i moon=75.864,45.169,0.717
2023-04-30 05:34:25.947Z cmds 20 CommandLink.py:121 > 2 29069 i obsMethod="Classical"
2023-04-30 05:34:25.947Z cmds 20 CommandLink.py:121 > 2 29069 i domeLights=0; topScreenPos=3.00,1.50,unknown
2023-04-30 05:34:25.947Z cmds 20 CommandLink.py:121 > 2 29069 i ringLampsStatus=9998,9998,9998,9998
2023-04-30 05:34:25.948Z cmds 20 CommandLink.py:121 > 2 29069 i ringLampsCmd=0.0,0.0,0.0,0.0
2023-04-30 05:34:25.948Z cmds 20 CommandLink.py:121 > 2 29069 i ringLamps=0.0,0.0,0.0,0.0
2023-04-30 05:34:25.948Z cmds 20 CommandLink.py:121 > 2 29069 i statusUpdate=92671,2,ag
2023-04-30 05:34:25.948Z cmds 20 CommandLink.py:121 > 2 29069 :
2023-04-30 05:34:31.115Z Gen2Cmd 20 Gen2Cmd.py:308 newGuideErrors:
Unknown macro: {'exposureId'}
2023-04-30 05:34:33.349Z cmdr 20 CmdrConnection.py:212 command returned ag 'acquire_field otf visit_id=0 exposure_time=3000'

Comment by chyan [ 23/Jun/23 ]

This problem is because we get the telescope information from `tel_status` and `env_condition`.

teleInfo = db.bulkSelect('tel_status','select pfs_visit_id, altitude, azimuth, insrot, adc_pa, m2_pos3 FROM tel_status '
                        f'ORDER BY pfs_visit_id DESC limit 1')

obsCond = db.bulkSelect('env_condition','select pfs_visit_id, outside_temperature, outside_pressure, outside_humidity '
                        f' FROM env_condition ORDER BY pfs_visit_id DESC limit 1')

As I know we use this actor by sending pfsVisitId=0 for not comsuming pfsVisitID. So, the key is that some actors need to update those information before everything

Comment by arnaud.lefur [ 08/Jul/23 ]

It might be actually a problem in agActor, since it's the one which calls updateTelStatus and fire agcc exposure.
Basically if ag doesn't wait on updateTelStatus to finish before starting agcc exposure then you have a problem.
Because the select from tel_status table in agcc is done early on cloomis Yoshida, Hiroshige does it make sense ?

Comment by arnaud.lefur [ 12/Jul/23 ]

Just got confirmation from Yoshida, Hiroshige that AG does not call updateTelStatus before hand, so it's up to agcc to ask for an update.
Also, I would certainly check that pfs_visit_id from tel_status table match the current agcc visit and throw at least a warning if it's not.

Comment by arnaud.lefur [ 15/Jul/23 ]

chyan so in agcc exposure command could you call updateTelStatus in the first place :

self.actor.cmdr.call(actor='gen2', cmdStr='updateTelStatus caller=agcc', timeLim=5.0)

I believe mcsActor does the same.

Comment by chyan [ 18/Jul/23 ]

arnaud.lefur Thank, I will update the software.

Comment by chyan [ 19/Jul/23 ]

issue fixed and close the ticket.

Generated at Sat Feb 10 16:41:42 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.

[INSTRM-1978] Check when telescope status in agc_exposure table is obtained Created: 12/May/23 Updated: 19/Jul/23 Resolved: 19/Jul/23