Rsyslog is the default logger for both Fedora and Debian - and if you're the default for those distros, you're also the default for CentOS, RHEL, Ubuntu, and dozens of other derivatives. So Rsyslog is probably the dominant Linux logging solution right now. But most people don't even know it - their Linux systems run, and they produce logs: no further thought required. But if you want to create new logs, or use Rsyslog's much vaunted remote-logging capability, you're going to need to understand the tool.

For me, this started when work and personal requirements popped up pretty much simultaneously. At work, we need federated logs across a cluster of machines that are all running the same service: it's been running surprisingly well so the need hasn't been pressing and we haven't had time - but having logs spread across several machines makes our occasional fits of debugging quite painful. I also have a personal machine that's crashing about once a week and I suspect it's because of memory issues. (The main problem is that the machine is eight years old, but it's got an i7 processor and 16G of memory so it's still useful.) I decided that solving this personal problem would be the best path to learning Rsyslog and eventually solving the work problem.

Rsyslog is an old Unix tool. It was started in 2004, but it was extending a history of standardized logging tools ("syslog") going back another decade or two. What this means is that, like so many Unix tools, it's amazingly complex and loaded with solutions for edge cases that few people need. But it gets worse: about 2-3 years ago the author(s) significantly changed the style of the configuration file, so most of the answers you find online to any particular Rsyslog problem are in a config style that no longer applies (but how do you know that if you're new to Rsyslog?):

# old-style:
$IncludeConfig /etc/rsyslog.d/*.conf
# new-style:
include(file="/etc/rsyslog.d/*.conf" mode="optional")

Sometimes you can mix and match the styles. But I find this even more confusing because while the dollar sign no longer prefaces directives, it still prefaces certain types of variables (but not all of them). It's a weird and tricky mix.

Worse, the definitive documentation isn't the man page (which tells you the basics and then says to look at the HTML documentation) but rather the pages maintained by a company called Adiscon. This is because the primary author of Rsyslog, Rainer Gerhards, owns and operates Adiscon GmbH which sells "Powerful tools for log, system, and security management." I don't object to open source authors making a living - far from it. But what I do object to is them maintaining buggy documentation, and providing no point of contact to fix those bugs because they'd rather sell you their service. I've found several bugs in what is theoretically the "definitive documentation," and I'm not even into the complex topics yet - I'm not impressed.

All these problems combined mean that learning Rsyslog is tough - and I think some basic, current examples would be useful to the world. That's what I hope to create here. (Did you check the date of this blog entry? If it's more than a couple years old, you might be wise to check elsewhere.)

Pieces of the puzzle:

• create and manage a new local log through Rsyslog
• route the log entries through Rsyslog to the right destination: initially a local log file, but eventually a remote logging server
• set up a host as a logging server that opens a port and receives other machine log messages (to my surprise, this was possibly the easiest part)
• but sending clear-text logs over a network isn't a good idea: set up encryption so log messages are encrypted in flight

This has been a lot of work, and I'm not finished. I'll be breaking it up across several blog articles. And please keep in mind that - as with many of my blog entries - I'm learning as I go so there may be errors. I'll correct them as best I can, but feel free to email me to suggest fixes. Keep in mind what I said: "I'm not finished." I'll add further entries as I figure this out.

Next: A New Log