Encrypting remote syslog with TLS (SSL)

Log messages can be delivered to Papertrail using TLS-encrypted syslog over TCP, as well as over UDP. Papertrail also supports TCP without TLS, though it isn’t often used.

This page describes how to configure rsyslog or syslog-ng for encrypted logging. If you’re using remote_syslog2 rather than rsyslog or syslog-ng, its README contains TLS setup instructions.

When configuring logging on your first system, consider configuring plaintext logging, verifying that it works, and then changing to TLS.

rsyslog.conf

Download root certificates

Save https://papertrailapp.com/tools/papertrail-bundle.pem into /etc/papertrail-bundle.pem on the log sender. For example:

$ sudo curl -o /etc/papertrail-bundle.pem https://papertrailapp.com/tools/papertrail-bundle.pem

Its MD5 checksum is currently 2c43548519379c083d60dd9e84a1b724, but it is updated periodically. Do not hard-code the checksum in deployment scripts. (To ensure a repeatable result, embed the entire bundle instead.)

rsyslog trusts these root CA keys to validate the key presented by Papertrail, preventing man-in-the-middle attacks.

Add TLS configuration

The instructions below assume rsyslog 4.0+.

On many distros, also install the rsyslog-gnutls package (including CentOS, Fedora, Debian, and Ubuntu).

Starting with unencrypted logging, find the line that sends to Papertrail, similar to

*.*     @logsN.papertrailapp.com:XXXXX`

Above that line, paste:

$DefaultNetstreamDriverCAFile /etc/papertrail-bundle.pem # trust these CAs
$ActionSendStreamDriver gtls # use gtls netstream driver
$ActionSendStreamDriverMode 1 # require TLS
$ActionSendStreamDriverAuthMode x509/name # authenticate by hostname
$ActionSendStreamDriverPermittedPeer *.papertrailapp.com

See Download root certificates to download the required CA file. Finally, on the Papertrail destination line, change the @ before the hostname to @@ (2 at-signs), which tells rsyslog to use TCP. For example:

*.*     @@logs.papertrailapp.com

If the destination line used a different hostname and/or a port other than 514, update the values in configuration to match. For example, to log to logsN.papertrailapp.com on port XXXXX with TCP and TLS:

*.*     @@logsN.papertrailapp.com:XXXXX

Restart rsyslog

Restart rsyslog so it detects the TLS-over-TCP destination:

$ sudo /etc/init.d/rsyslog restart

After logging is working, we strongly recommend adding this configuration to make rsyslog queue locally and reconnect if the TCP connection drops.

Without this extra configuration, rsyslog may not reconnect to Papertrail or may block on inbound syslog() calls from apps, both of which are bad. Head over here and paste it in.

Troubleshooting

could not load module '/usr/lib/rsyslog/lmnsd_gtls.so',
rsyslog error -2078 [try http://www.rsyslog.com/e/2068 ]

First, make sure that module exists by running ls against the path in the error, such as:

$ ls -la /usr/lib/rsyslog/lmnsd_gtls.so

If it doesn’t exist, install the related package (often called rsyslog-gnutls) or if you compiled rsyslog from source, compile the module.

Second, ensure that the user that runs rsyslog has permissions to read Papertrail’s public key (in the instructions above, /etc/papertrail-bundle.pem). On many distributions, rsyslog starts as root and then drops to a user. In that case, run: chmod 644 /etc/papertrail-bundle.pem to let all users read the key file.

Finally, this may appear if you are using $ModLoad lmnsd_gtls to explicitly load the TLS module, and that configuration option occurs before the $DefaultNetstreamDriverCAFile has been defined. Explicitly loading the module is rarely required and the configuration above does not use it. We recommend removing that $ModLoad lmnsd_gtls option and relying on autoloading. If your lmnsd_gtls needs to be explicitly loaded, such as because it is in a non-default location, move the $DefaultNetstreamDriverCAFile config line above the $ModLoad line.

For more generic troubleshooting information, see Troubleshooting remote syslog reachability.

For additional logging tips, see Advanced Unix logging tips, notably Aggregate local log files with rsyslog and Tweak queue options for connection failure.

syslog-ng.conf

Download root certificates

Download and extract root CA certificates for syslog-ng:

$ sudo mkdir /etc/syslog-ng/cert.d
$ cd /etc/syslog-ng/cert.d
$ curl https://papertrailapp.com/tools/papertrail-bundle.tar.gz | sudo tar xzf -

The MD5 checksum of papertrail-bundle.tar.gz is currently 23d82ae8698d41e75d1e85224d6a9ad5, but it is updated periodically. Do not hard-code the checksum in deployment scripts. (To ensure a repeatable result, embed the entire bundle instead.)

syslog-ng trusts these root CA certificates to validate the authenticity of the key presented by Papertrail, preventing man-in-the-middle attacks.

Add TLS configuration

Starting with unencrypted logging, find the line that sends to Papertrail. It should be in the format: destination d_papertrail { .. }.

In that stanza, remove the existing udp(..) line. Replace it with the new TLS-over-TCP destination:

tcp("logs.papertrailapp.com" port(514) tls(ca_dir("/etc/syslog-ng/cert.d")) );

If the udp configuration used a different hostname and/or a port other than 514, update the values in the new tcp configuration to match. For example, to log to logsN.papertrailapp.com on port XXXXX with TCP and TLS:

destination d_papertrail {
  tcp("logsN.papertrailapp.com" port(XXXXX) tls(ca_dir("/etc/syslog-ng/cert.d")) );
};

Restart syslog-ng

Restart syslog-ng so it detects the TLS-over-TCP destination:

$ sudo killall -HUP syslog-ng

Verify (recommended)

To verify that messages are encrypted, run a packet sniffer like tcpdump, generate a log message, and confirm that the cleartext body is not shown. For example, to output the payload of packets to logsN.papertrailapp.com:

$ sudo tcpdump -s 1500 -X src or dst logsN.papertrailapp.com

Change logsN to your Papertrail host, as shown under Log Destinations. You should see packets flowing, and they should not contain human-readable log text.

Troubleshooting

Consider setting up cleartext logging first, then moving to TLS encryption once cleartext works.

If cleartext logging is working, but TLS is not, attempt a TLS-encrypted TCP connection to Papertrail using the s_client feature of openssl.

Run:

$ openssl s_client -showcerts -connect logsN.papertrailapp.com:XXXXX -CAfile /etc/papertrail-bundle.pem

where logsN and XXXXX are the name and port number shown under log destinations and /etc/papertrail-bundle.pem is the path to the certificate that can be downloaded here.

For more generic troubleshooting information, see Troubleshooting remote syslog reachability.