Configuring centralized logging from systemd

To send system logs from Linux distributions using systemd (including newer releases of CentOS and CoreOS), follow the steps shown below to forward output from journalctl to Papertrail.

For applications running in a Docker container, the options described on the Docker configuration page are recommended, but also see Docker and systemd below.

Forwarding system logs

Create a new unit/service that pipes output from journalctl into ncat or socat by following the steps below:

  • Create a new unit file named papertrail.service under /etc/systemd/system (root permissions required).
  • Paste the directives shown below, and replacing logsN and XXXXX with the values that appear under log destinations.
[Unit]
Description=Papertrail
After=systemd-journald.service
Requires=systemd-journald.service

[Service]
ExecStart=/bin/sh -c "journalctl -f | ncat --ssl logsN.papertrailapp.com XXXXX"
TimeoutStartSec=0
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target
  • Run $ sudo systemctl enable /etc/systemd/system/papertrail.service to enable the unit.
  • Then run $ sudo systemctl start papertrail.service to start it.
  • Finally, use $ journalctl -f -u papertrail.service to view the local logs and ensure the new unit has started successfully

Using socat

If ncat is not available, use socat instead by:

ExecStart=/bin/sh -c "journalctl -f | socat - SSL:logsN.papertrailapp.com:XXXXX,cafile=/etc/papertrail-bundle.pem"

Again, replace logsN and XXXXX with the values that appear under log destinations.

Docker and systemd

When using systemd to send logs from Docker containers, as on CoreOS, the solutions above will result in all Docker logs coming in under the program name docker rather than the service name. If sending logs from containers via systemd is desirable, try the solution created by Papertrail customer C2FO:

journalctl -o json -f | jq -r --unbuffered '"\(.__REALTIME_TIMESTAMP | tonumber / 1000000) \(._HOSTNAME) \(._SYSTEMD_UNIT // .UNIT // .SYSLOG_IDENTIFIER): \(.msgtype = (.MESSAGE | type) | if .msgtype == "array" then (.MESSAGE | implode) else .MESSAGE end)"' | awk '{ $1=strftime("%b %d %T", $1); print $0; fflush(); }'

piped to ncat or socat. This is easiest to place in a separate script, but it can also be placed directly in an ExecStart directive. It makes use of the JSON format for journalctl logs, transformed to a syslog string using jq and awk. There’s also an option to set up coproc to better detect sending failures.