https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/INSTRM-38 Instrument control development <p>We need to make sure that drops in NTP service to any part of the instrument or any divergences are noted and reported.</p> INSTRM-38

Add timebase (NTP/PTP) monitoring

Task Major Open Unresolved Unassigned cloomis Fri, 16 Dec 2016 17:35:33 +0000 Fri, 31 Mar 2023 16:41:12 +0000 0 2 <p>I don't if my understanding is correct, but basically on the PFS network we have one machine which is synchronised with the outside world every (hours ? days ?) . Then every clients on our network synchronise with that machine right ?.</p> <p>What do you want to monitor exactly ? the time drift ? if the ntp server is unreachable ?</p> <p>From system health monitoring point of view, we should monitor both whether NTP sync is working at each control system or not, and also how large their drift is.<br/> This might not be monitored from actor layer, but at least shall be handled by OS monitoring (or layer above than it).</p> <p>I've installed <b>ntplib</b> on our anaconda distribution. I've tested a bit and it seems to answer our expectations.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=hiro" class="user-hover" rel="hiro">Yoshida, Hiroshige</a> Does subaru alert on ntp failures?</p> <p>This actually just hit us. Because we rebooted several BEEs yesterday it happened to be obvious: they have no batteries and so came up 21 years off. But since the instrument NTP server was gone all of the other system clocks would have started drifting, along with the dates of and in the various image files and logs. It might not take too many days for the agc and agcc computers to drift off by a few seconds, which could have been deeply confusing. </p> <p>Is this a place for prometheus, etc? And/or a second NTP server? Thoughts, <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=hiro" class="user-hover" rel="hiro">Yoshida, Hiroshige</a>?</p> Development Rank 0|ii019z: