Encrypting remote syslog with TLS (SSL)

Log messages can be delivered to Papertrail using TLS-encrypted syslog over TCP. Here’s how to configure rsyslog and syslog-ng for encrypted logging.

Papertrail accepts messages with UDP as well as TCP with TLS. Papertrail also supports TCP without TLS, though it is rarely relevant.

If you’re using remote_syslog2 rather than rsyslog or syslog-ng, its README contains TLS setup instructions.

Note: When configuring logging on your first system, consider configuring plaintext logging, verifying that it works, and then changing to TLS.

Choose syslog daemon

Next, go to the configuration instructions for your syslog daemon: rsyslog or syslog-ng

Configuration: rsyslog.conf

1. Download root certificates

Save https://papertrailapp.com/tools/papertrail-bundle.pem into /etc/papertrail-bundle.pem on the log sender. For example:

sudo curl -o /etc/papertrail-bundle.pem https://papertrailapp.com/tools/papertrail-bundle.pem

Its MD5 checksum is ba3b40a34ec33ac0869fa5b17a0c80fc.

rsyslog trusts these root CA keys to validate the key presented by Papertrail, preventing man-in-the-middle attacks.

2. Add TLS configuration

Important: When using TLS, we strongly recommend running the latest minor version of your rsyslog release. Where possible, run rsyslog v7 or v8; these fix many bugs with TLS. The instruction below assume rsyslog 4.0+.

On many distros, also install the rsyslog-gnutls package (including CentOS, Fedora, Debian, and Ubuntu).

Starting with unencrypted logging, find the line which sends to Papertrail. The line should be similar to: *.* @<host>.papertrailapp.com.

Above that line, paste:

$DefaultNetstreamDriverCAFile /etc/papertrail-bundle.pem # trust these CAs
$ActionSendStreamDriver gtls # use gtls netstream driver
$ActionSendStreamDriverMode 1 # require TLS
$ActionSendStreamDriverAuthMode x509/name # authenticate by hostname
$ActionSendStreamDriverPermittedPeer *.papertrailapp.com

See above to download the required CA file. Finally, on the Papertrail destination line, change the @ before the hostname to @@ (2 at-signs), which tells rsyslog to use TCP. For example:

*.*                                         @@logs.papertrailapp.com

If the destination line used a different hostname and/or a port other than 514, update the values in configuration to match. For example, to log to logs2.papertrailapp.com on port 1111 with TCP and TLS:

*.*                                         @@logs2.papertrailapp.com:1111

3. Restart

Restart rsyslog so it detects the TLS-over-TCP destination:

sudo /etc/init.d/rsyslog restart

Important: After logging is working, we strongly recommend adding this configuration to make rsyslog queue locally and reconnect if the TCP connection drops.

Without this extra configuration, rsyslog may not reconnect to Papertrail or may block on inbound syslog() calls from apps, both of which are bad. Head over here and paste it in.

Troubleshooting

could not load module '/usr/lib/rsyslog/lmnsd_gtls.so',
rsyslog error -2078 [try http://www.rsyslog.com/e/2068 ]

First, make sure that module actually exists by running ls against the path in the error, such as:

ls -la /usr/lib/rsyslog/lmnsd_gtls.so

If it doesn’t exist, install the related package (often called rsyslog-gnutls) or if you compiled rsyslog from source, compile the module.

Second, ensure that the user which runs rsyslog has permissions to read Papertrail’s public key (in the instructions above, /etc/papertrail-bundle.pem). On many distributions, rsyslog starts as root and then drops to a user. In that case, run: chmod 644 /etc/papertrail-bundle.pem to let all users read the key file.

Finally, this may appear if you are using $ModLoad lmnsd_gtls to explicitly load the TLS module, and that configuration option occurs before the $DefaultNetstreamDriverCAFile has been defined. Explicitly loading the module is rarely required and the configuration above does not use it. We recommend removing that $ModLoad lmnsd_gtls option and relying on autoloading. If your lmnsd_gtls needs to be explicitly loaded, like because it is in a non-default location, move the $DefaultNetstreamDriverCAFile config line above the $ModLoad line.

For more generic troubleshooting information, see Troubleshooting remote syslog reachability.

3. Optional: Monitor local files, tweak queue settings

To have rsyslog read local app log files (instead of using remote_syslog2) or adjust how it behaves during a TCP connection failure, see Advanced Unix logging tips, specifically Aggregate local log files with rsyslog and Tweak queue options for connection failure.

Configuration: syslog-ng.conf

1. Download root certificates

Download and extract root CA certificates for syslog-ng:

sudo mkdir /etc/syslog-ng/cert.d
cd /etc/syslog-ng/cert.d
curl https://papertrailapp.com/tools/papertrail-bundle.tar.gz | sudo tar xzf -

The MD5 checksum of papertrail-bundle.tar.gz is 2782349be167b25355537c2ce2b395e7.

Syslog-ng trusts these root CA certificates to validate the authenticity of the key presented by Papertrail, preventing man-in-the-middle attacks.

2. Add TLS configuration

Starting with unencrypted logging, find the line that sends to Papertrail. It should be in the format: destination d_papertrail { .. }.

In that stanza, remove the existing udp(..) line. Replace it with the new TLS-over-TCP destination:

tcp("logs.papertrailapp.com" port(514) tls(ca_dir("/etc/syslog-ng/cert.d")) );

If the udp configuration used a different hostname and/or a port other than 514, update the values in the new tcp configuration to match. For example, to log to logs2.papertrailapp.com on port 1111 with TCP and TLS:

destination d_papertrail {
  tcp("logs2.papertrailapp.com" port(1111) tls(ca_dir("/etc/syslog-ng/cert.d")) );
};

3. Restart

Restart syslog-ng so it detects the TLS-over-TCP destination:

sudo killall -HUP syslog-ng

Verify (optional, recommended)

To verify that messages are encrypted, run a packet sniffer like tcpdump, generate a log message, and confirm that the cleartext body is not shown. For example, to output the payload of packets to logs.papertrailapp.com:

sudo tcpdump -s 1500 -X src or dst logs.papertrailapp.com

You should see packets flowing and they should not contain human-readable log text.

Troubleshooting

First, consider setting up cleartext logging first, then moving to TLS encryption once cleartext works.

Second, attempt a TLS-encrypted TCP connection to Papertrail using the s_client feature of openssl. Run:

openssl s_client -showcerts -connect <host>.papertrailapp.com:11111 -CAfile /etc/papertrail-bundle.pem

where <host> and 11111 are the name and port number shown under log destinations and /etc/papertrail-bundle.pem is the path to the certificate that can be downloaded here.

For more generic troubleshooting information, see Troubleshooting remote syslog reachability.