The ELK Stack with Beats: Securing the Beats-to-Logstash Connection

Read the first item in this Table of Contents if you haven't been here before.

Table of Contents

I meant to make the blog entry about Filebeat just one part, but it was running long and I realized I still had a lot to cover about securing the connection. This isn't a stand-alone entry: you'll need to read the previous entry and do the groundwork for the insecure Beats before you can proceed to securing it.

Securing the connection turns out to be somewhat complex: some of it will be familiar to anyone who's dealt with SSL/TLS certificates before, but this adds some odd twists to the process. Essentially, the server (in our context, logstash, I'm using the hostname "elktest") needs to have a TLS certificate, and the client(s) (filebeat, hostname "beatbox") needs to know how to talk to that certificate. There are two ways to do that: have a legitimate cert from a real certificate authority, and filebeat will know about the cert because the operating system says the cert is legit through the regular channels. If you go self-signed (which is essentially the only way if you're running a clump of virtual machines for testing, and the way I would recommend anyway), you have to specifically inform filebeat of this fact in its config file. And here's one of the ugly bits: while filebeat is now equipped to properly communicate encrypted logs to logstash, logstash doesn't authenticate its incoming connections and may well accept any valid incoming Beats data. While this is a rather bizarre and unlikely security threat (ie. someone else spamming your ELK server with Beats data), it seems a hole worth plugging because it could really, seriously mess up your data. Possible solutions include adding firewall control of incoming filebeat data to the TLS authentication, or - better - a VPN, with the logstash server only listening to the VPN.

On the server (logstash/elasticsearch) create the cert and associated files:

openssl req -subj /CN=elktest -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout elktest.logstash.key -out elktest.logstash.crt

Change the two occurrences of my hostname "elktest" to your own wherever it occurs.

If you're not familiar with SSL/TLS certificate generation, you can see my intro to the subject or you can do it by rote and trust that I'm giving you something that'll work. I wouldn't recommend this: while I believe my instructions are valid, there are numerous reasons you should actually understand this process. If you have to change it - now, or especially later - you'll need to know what to change, and that requires knowledge of the process.

Copy the .crt file to the filebeat machine. As the certificate is considered public knowledge, I stored it in /etc/filebeat/elktest.logstash.crt. To use this, edit the "output:" stanza of /etc/filebeat/filebeat.yml:

output:
  logstash:
    protocol: https
    hosts: ["elktest:5044"]
    tls:
      certificate_authorities: [ "/etc/filebeat/elktest.logstash.crt" ]

This is an addition of two lines (the last two): the rest of the file remains the same. If you restart filebeat (and you haven't reconfigured logstash yet), you'll find filebeat may actually do some logging:

Mar 10 13:06:11 beatbox /usr/bin/filebeat[3895]: transport.go:125: SSL client failed to connect with: EOF

So apparently it's trying to speak TLS and logstash isn't understanding it. This seems to appear on start-up and every minute or so thereafter. Let's fix that.

On the ELK machine, copy the certificate and key to somewhere in /etc/ - I chose /etc/logstash/ - and then change to that directory and run chmod 600 elktest.logstash.key. Now edit /etc/logstash/conf.d/beats.conf:

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/logstash/elktest.logstash.crt"
    ssl_key => "/etc/logstash/elktest.logstash.key"
  }
}

Three new lines (ssl, ssl_certificate and ssl_key), with the rest of the file remaining the same. Restart logstash.

With my 600 key I'm getting a permission error :

Permission denied - /etc/logstash/elktest.logstash.key

This appears in both the log and the stdout file, but not the error file, go figure. But this is because logstash runs as user logstash, and the simplest solution (although its security is dubious - this is why I recommend using self-signed, not a legit certificate of value) is to do this: chown logstash /etc/logstash/elktest.logstash.key. After this change logstash restarts cleanly and filebeat starts sending data to it as expected.

As a test, I switched filebeat back to unencrypted communication, and it seems to be refusing to connect - or logstash is refusing to accept the data. The logging on both ends is non-existent, so I'm not sure what's going on. But at least there's some hope you won't be spammed by mistaken connections that don't have the right certificate or aren't encrypting.

Better yet, if filebeat fails to deliver (in this case because I deliberately crippled it), it will resend older log entries later when it's able to. I haven't tested the extent of this (I saw it retransmit perhaps 1KB of data, not exactly a lot), but at least there's something.

This behaviour suggests that the certificate should be treated as a "secret": an attacker would appear to need to possess it to spam your ELK/logstash server. A certificate is usually considered public information, but one generated just for this purpose could be secured.