[INSTRM-38] Add timebase (NTP/PTP) monitoring Created: 17/Dec/16  Updated: 01/Apr/23

Status: Open
Project: Instrument control development
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major
Reporter: cloomis Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

We need to make sure that drops in NTP service to any part of the instrument or any divergences are noted and reported.



 Comments   
Comment by arnaud.lefur [ 18/May/17 ]

I don't if my understanding is correct, but basically on the PFS network we have one machine which is synchronised with the outside world every (hours ? days ?) . Then every clients on our network synchronise with that machine right ?.

What do you want to monitor exactly ? the time drift ? if the ntp server is unreachable ?

Comment by shimono [ 19/May/17 ]

From system health monitoring point of view, we should monitor both whether NTP sync is working at each control system or not, and also how large their drift is.
This might not be monitored from actor layer, but at least shall be handled by OS monitoring (or layer above than it).

Comment by arnaud.lefur [ 16/Jun/17 ]

I've installed ntplib on our anaconda distribution. I've tested a bit and it seems to answer our expectations.

Comment by arnaud.lefur [ 18/Mar/23 ]

Yoshida, Hiroshige Does subaru alert on ntp failures?

Comment by cloomis [ 01/Apr/23 ]

This actually just hit us. Because we rebooted several BEEs yesterday it happened to be obvious: they have no batteries and so came up 21 years off. But since the instrument NTP server was gone all of the other system clocks would have started drifting, along with the dates of and in the various image files and logs. It might not take too many days for the agc and agcc computers to drift off by a few seconds, which could have been deeply confusing.

Is this a place for prometheus, etc? And/or a second NTP server? Thoughts, Yoshida, Hiroshige?

Generated at Sat Feb 10 16:20:51 JST 2024 using Jira 8.3.4#803005-sha1:1f96e09b3c60279a408a2ae47be3c745f571388b.