arnaud.lefur to ask cloomis for clarification.

No further inputs needed from cloomis.

Important for temperature boards - logging goes to a local file.

rsyslog is as disgusting an accretion of idiosyncratic features as I have ever seen. But it is well tested and does provide the features we need. Specifically, python logging writes to "syslog" handlers, rsyslog lets you add memory or disk queues on outputs (solving INSTRM-753), and rsyslog writes to files, databases, streams of every kind.

Just to corral the craziness, I'll always use RFC5424 ("new syslog" format; the RFC is written by the rsyslog author, sigh), and will always use the "advanced" configuration/scripting language.

And will write to per-day log files saved in per-actor directories. Rsyslog allows writing to every imaginable output; we might well add PostgreSQL at some point, but not for now.

Bump, damnit. At least for the temperature boards: the disk filled up on temps-b1. We don't need to record all command traffic, but I think we should.

bump bump on the temps boards. There is no workaround besides not logging, and we are wedging systems.

Arnaud to look at Craig's proposal

Following today's SMx discussion, this could be the cause of the recent problem where the STS lost contact with SM1.

Bump. On 2023-07-26 the /data NFS mount just got bounced on all the PFS hosts, and python logging from all the PFS actors to /data/logs stopped. Being the scenes the failure might have been worse than mere silence: where were all the messages going, I wonder? Getting queued, but is there a limit, etc.

On the original question about whether the ugly but proven rsyslog is the right tool? I will not look further but invite others to do so. It is proven and we have proven that we can use it. We would been to choose a logging host, select the output (files vs db vs xxx) and write the exploder. Thoughts on the host, Yoshida, Hiroshige?

Bump again, at least for the SPS temp hosts, which do not use NFS at all and have one tiny disk.

Yoshida, Hiroshige we would need a rsyslog host, is that something you / CDM could easily provide ?

Can't we use the existing logger-ics? It currently runs rsyslog (mainly for collecting server/host messages).

Is there a VM at IRx? Would be nice to maintain writes for SPS during outages. Could just use mhs-ics I guess. Maybe that's the right move.

Also, does anyone have thoughts about storage? The obvious implementation is to explode back into the same /data/logs tree, but I'm wondering if anyone has any experience/preference with any other. Are we running any log-friendly backends?