https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/INSTRM-322 Instrument control development <p>All logging for actors goes via <tt>opscore.sdss3logging</tt>, which implements a StreamHandler to a RotatingLogHandler. This will not scale well to PFS: we should replace the final handler with a SyslogHandler to the instrument's rsyslog server.</p> INSTRM-322

Switch actor logging to rsyslog.

Task Normal Open Unresolved arnaud.lefur cloomis SPS Tue, 10 Apr 2018 14:05:53 +0000 Fri, 10 Nov 2023 19:13:46 +0000 tron_actorcore 0 4 <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=arnaud.lefur" class="user-hover" rel="arnaud.lefur">arnaud.lefur</a> to ask <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a> for clarification.</p> <p>No further inputs needed from <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=cloomis" class="user-hover" rel="cloomis">cloomis</a>. </p> <p>Important for temperature boards - logging goes to a local file.</p> <p>rsyslog is as disgusting an accretion of idiosyncratic features as I have ever seen. But it is well tested and does provide the features we need. Specifically, python <tt>logging</tt> writes to "syslog" handlers, rsyslog lets you add memory or disk queues on outputs (solving <a href="https://pfspipe.ipmu.jp/jira/browse/INSTRM-753" title="Add short log cache to local disk for BEEs and SMBs" class="issue-link" data-issue-key="INSTRM-753"><del>INSTRM-753</del></a>), and rsyslog writes to files, databases, streams of every kind.</p> <p>Just to corral the craziness, I'll always use RFC5424 ("new syslog" format; the RFC is written by the rsyslog author, sigh), and will always use the "advanced" configuration/scripting language.</p> <p>And will write to per-day log files saved in per-actor directories. Rsyslog allows writing to every imaginable output; we might well add PostgreSQL at some point, but not for now.</p> <p>Bump, damnit. At least for the temperature boards: the disk filled up on temps-b1. We don't <em>need</em> to record all command traffic, but I think we should.</p> <p>bump bump on the temps boards. There is no workaround besides not logging, and we are wedging systems.</p> <p>Arnaud to look at Craig's proposal</p> <p>Following today's SMx discussion, this could be the cause of the recent problem where the STS lost contact with SM1.</p> <p>Bump. On 2023-07-26 the /data NFS mount just got bounced on all the PFS hosts, and python logging from <b>all</b> the PFS actors to /data/logs stopped. Being the scenes the failure might have been worse than mere silence: where were all the messages <b>going</b>, I wonder? Getting queued, but is there a limit, etc.</p> <p>On the original question about whether the ugly but proven rsyslog is the right tool? I will not look further but invite others to do so. It is proven and we have proven that we can use it. We would been to choose a logging host, select the output (files vs db vs xxx) and write the exploder. Thoughts on the host, <a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=hiro" class="user-hover" rel="hiro">Yoshida, Hiroshige</a>?</p> <p>Bump again, at least for the SPS temp hosts, which do not use NFS at all and have one tiny disk.</p> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=hiro" class="user-hover" rel="hiro">Yoshida, Hiroshige</a> we would need a rsyslog host, is that something you / CDM could easily provide ?</p> <p>Can't we use the existing logger-ics? It currently runs rsyslog (mainly for collecting server/host messages).</p> <p>Is there a VM at IRx? Would be nice to maintain writes for SPS during outages. Could just use mhs-ics I guess. Maybe that's the right move.</p> <p>Also, does anyone have thoughts about storage? The obvious implementation is to explode back into the same /data/logs tree, but I'm wondering if anyone has any experience/preference with any other. Are we running any log-friendly backends? </p> Blocks INSTRM-753 Relates INSTRM-2045 Development Rank 0|zzst5b:2 Sprint SM1-2019 E SM1-2019 J SM1PD-2020 F SM1PD-2020 J Story Points 6.0