Troubleshooting remote syslog reachability

How to test reachability, plus message length and protocol notes.



After configuring a system to log to Papertrail, you may wish to verify end-to-end reachability from your system to Papertrail (without making any system changes). This can identify whether a problem is with the system/logger configuration, network reachability, or Papertrail configuration.

We’re here to help and we love this stuff, so don’t be shy (chat and email here).

Message Length


Papertrail accepts UDP packets of up to 1024 bytes (1 KB) in total packet length, which is about 950 characters for the log message text. That’s a very long line - 5-10 lines of typical dense text.

Note that the limit is applied separately to each packet (log line). Almost all lines longer than 1 KB are spread across multiple lines anyway (for example, tracebacks). As a result, this limit is almost never encountered.

1024 bytes is a protocol requirement defined in RFC 3164, which nearly all senders enforce (either by dropping longer messages or splitting them into multiple conforming messages). Even without the RFC requirement, UDP messages would be constrained by the Internet MTU of 1500 bytes minus the UDP header.


Papertrail accepts messages over TCP that are up to 100,000 bytes (~100 KB) long. 100 KB was chosen as the outer edge of what we could imagine a use for and to ensure Papertrail can handle messages from default configurations of all common syslog daemons.

Message Format

Papertrail accepts messages which are formatted according to either RFC 3164 or RFC 5424. Senders may use either or both. No configuration within Papertrail is necessary.

While RFC 5424 defines the hostname field to be up to 255 characters, Papertrail truncates the hostname value to 120 characters.

Papertrail also accepts many types of malformed messages, generally because the message formats are generated by common-used software and hardware.

Sending over TCP

By default, Papertrail accepts messages over TCP with TLS. Unencrypted TCP is also supported, but not enabled by default. To enable it, visit Destinations.

Test: Basic IP Reachability

To see whether your system can reach Papertrail, run:

telnet 514

You should see a connection established (“Connected to..”) but will not see any output. Here’s an example:

$ telnet 514

Connected to
Escape character is '^]'.
telnet> close

Note that this only tests TCP reachability to the default syslog port. It does not test UDP reachability.

Test: Hostname Format

The syslog packet includes a hostname field. Papertrail accepts syslog messages with hostnames that can exist in DNS (letters, numbers, periods, hyphens, and to be permissive, underscores).

Occasionally the operating system will have an invalid hostname like (none) or the syslog daemon will be transmitting a hostname value different from the operating system hostname.

OS hostname


hostname -f

Check the output to verify that the system hostname doesn’t contain parentheses or other non-DNS characters.

Syslog hostname

Using syslog-ng? Look for the chain_hostnames option in syslog-ng.conf. If found, comment out that line, run /etc/init.d/syslog-ng restart, and retry the test.

This will ensure that syslog-ng does not transmit hostnameA@hostnameB as the hostname. (If you use chained hostnames, contact us or consider keep_hostname).

Test: Sending data

To see whether your logger(s) are actually sending packets to Papertrail, run:

sudo tcpdump -n -s 1500 -X port 11111

.. where 11111 is the port that the logger is sending to Papertrail on. While tcpdump is running, generate some log messages. You should see roughly one packet in the tcpdump output for each log message.

The output from tcpdump may be extremely verbose. To output it to a file, such as to send to Papertrail support, run:

sudo tcpdump -n -s 1500 -X port 11111 -w tcpdump.pcap

Test: Examine logger daemon activity

To see the actual behavior of the logger daemon (like rsyslog or remote_syslog2), use strace. For example:

sudo strace -s 500 -tfp 22222

.. where 22222 is the process ID of the logger, as seen in the second column of ps auxw. While strace is running, generate some log messages. The output from strace may be extremely verbose. To output it to a file, such as to send to Papertrail support, run:

sudo strace -s 500 -tfp 22222 -o strace.log

If the strace program is not found, install your distribution’s package with a command like sudo yum install strace or sudo apt-get install strace.

Test: System Logging Config

To generate a log message that uses your system syslogd config (/etc/rsyslog.conf, /etc/syslog-ng.conf, or another syslog config), run:

logger "test"

That will generate a test message containing “test” as a syslog message.

Note: if your system syslog is not setup properly (or cannot communicate with the remote syslog destination), this test message will also be affected by that problem. Try the instructions below to send a message directly to a host & port, without using the system’s syslog.

Test: Use remote_syslog2

Install remote_syslog2, create a placeholder log file, then tell remote_syslog2 to watch it and send new lines to a hostname/port provided by Papertrail, and not to daemonize. Example for <host>

touch /tmp/testfile
remote_syslog -d <host> -p 12345 -D /tmp/testfile

Then in another terminal or with remote_syslog2 backgrounded, append lines to the file:

echo "This is a test message" >> /tmp/testfile

Each message should appear on Papertrail (or your syslog destination) momentarily.

Stop remote_syslog2 with Ctrl-C when finished testing.

Interpreting tests

The tests above are intended to help guide to the likely cause. Choose the test result below.

Test message arrived

The problem is probably with system or app server log configuration. Start here.

For example, consider running the log sender in debug mode and in the foreground to look for error messages, such as rsyslog -dn or syslog-ng -dF.

Message arrived but was attributed to incorrect system

The problem is probably with the name which the sender is transmitting to Papertrail (and which Papertrail uses to identify to sender). Contact us and we’ll make specific suggestions for your hosting environment.

Test message did not arrive

There are a few possible causes:

I’m not sure

Say hello and we’ll help. Please include anything you’ve already tried.

Firewall policies

To add an explicit firewall rule with iptables that allows outgoing traffic based on a destination port (recommended), run these 2 commands. Change 514 to to the destination port:

iptables -I OUTPUT 1 --protocol udp --dport 514 -j ACCEPT
iptables -I OUTPUT 1 --protocol tcp --dport 514 -j ACCEPT

To allow based on destination IP, add rules allowing traffic to,, and

iptables -I OUTPUT 1 -d -j ACCEPT
iptables -I OUTPUT 1 -d -j ACCEPT
iptables -I OUTPUT 1 -d -j ACCEPT

…then generate another test message. If this works, make sure to save these firewall rules for the next reboot.

When using IP subnet-based egress firewall rules, please sign up for Papertrail’s log destination changes mailing list. This is only necessary if your network configuration requires you to know about every new or deprecated log destination subnet. It’s not necessary if you use a current IP as your log destination only because the sender doesn’t support sending to a domain. Papertrail tracks inbound messages and will email account contacts before removing any destinations, so you’ll always hear from us if you need to change an existing IP.


If your system enforces SELinux policies and the destination port is not 514, you may need to add the port to the set of ports which syslogd can open network connections to.

To confirm that SELinux is the cause, execute strace -tt -s 500 -fp PID -o strace.log (replace PID with the pid of your logging software), and then grep for EACCESS:

grep EACCESS strace.log

Output similar to the following will indicate that a policy change is required:

connect(6, {sa_family=AF_INET, sin_port=htons(XXXX), sin_addr=inet_addr("67.214.X.X")}, 16) = -1 EACCES (Permission denied)

To permit access, on RHEL6:

yum install policycoreutils-python
semanage port -a -t syslogd_port_t -p tcp 12345

To see current port settings:

semanage port --list

Idle timeouts

Papertrail automatically closes idle TCP connections after 15 minutes of inactivity. Log senders should already handle this as part of a full implementation, since connections are routinely affected by Internet reachability changes. Senders can elect to immediately reconnect, or leave the connection closed until new log messages are available.

In addition to Papertrail’s timeout, Google Compute Engine and Microsoft Azure also have their own.

Rate limits

The following rate limits apply for log messages transmitted to Papertrail:

For UDP:

For TCP:

For more information on why they were introduced, see Introducing Syslog Rate Limits.

Hitting these limits? Let us know. Please include the logger name and associated configuration file.

Juniper/Netscreen ScreenOS

To allow log traffic to pass through the firewall, configure it via the Web interface as follows:

Creating a Custom Service

Applying the Service

GNU syslogd


Until recently, a few distributions (CentOS, Gentoo) provided GNU syslogd rather than syslog-ng or rsyslog. GNU syslogd only supports logging to the default syslog port (514), not user-selected destination ports. To see whether you run GNU syslogd, check:

If so, this may be the problem.


Papertrail has a few workarounds for GNU syslogd. Choose one:

A. Tell Papertrail your system’s IP address so that it can log to the default port.

To do that, visit Add Systems and click the Alternatives link near the top. On the Alternatives page, fill in the IP address of your system. Papertrail will provide a log destination which uses the default syslog port and will work with GNU syslogd.

B. Use the standalone remote_syslog2 log collector to send to Papertrail.

remote_syslog2 can send both app and system log files. It doesn’t modify the system-wide logging configuration at all and Papertrail does not need the IP addresses of your servers. Setup instructions.

C. Upgrade to syslog-ng or rsyslog.

These are better loggers by any measure. CentOS and Gentoo were the last distributions to switch.