Papertrail Knowledge Base

Troubleshooting remote syslog reachability

How to test reachability, plus message length and protocol notes.

Purpose

Send test log messages to Papertrail using your system syslog configuration or a standalone sender (plus some troubleshooting suggestions).

After configuring a system to log to Papertrail, you may wish to verify end-to-end reachability from your system to Papertrail (without making any system changes). This can identify whether a problem is with the system/logger configuration, network reachability, or Papertrail configuration.

We're here to help and we love this stuff, so don't be shy (chat and email here).

Message Length

UDP

Papertrail accepts UDP packets of up to 1024 bytes (1 KB) in total packet length, which is about 950 characters for the log message text. That's a very long line - 5-10 lines of typical dense text.

Note that the limit is applied separately to each packet (log line). Almost all lines longer than 1 KB are spread across multiple lines anyway (for example, tracebacks). As a result, this limit is almost never encountered.

1024 bytes is a protocol requirement defined in RFC 3164, which nearly all senders enforce (either by dropping longer messages or splitting them into multiple conforming messages). Even without the RFC requirement, UDP messages would be constrained by the Internet MTU of 1500 bytes minus the UDP header.

TCP

Papertrail accepts messages over TCP that are up to 100,000 bytes (~100 KB) long. 100 KB was chosen as the outer edge of what we could imagine a use for and to ensure Papertrail can handle messages from default configurations of all common syslog daemons.

Sending over TCP

By default, Papertrail accepts messages over TCP with TLS. Unencrypted TCP is also supported, but not enabled by default. To enable it, visit Destinations.

GNU syslogd

Test

Until recently, a few distributions (CentOS, Gentoo) provided GNU syslogd rather than syslog-ng or rsyslog. GNU syslogd only supports logging to the default syslog port (514), not user-selected destination ports. To see whether you run GNU syslogd, check:

If so, this may be the problem.

Workarounds

Papertrail has a few workarounds for GNU syslogd. Choose one:

A. Tell Papertrail your system's IP address so that it can log to the default port.

To do that, visit Add Systems and click the Alternatives link near the top. On the Alternatives page, fill in the IP address of your system. Papertrail will provide a log destination which uses the default syslog port and will work with GNU syslogd.

B. Use the standalone remote_syslog log collector to send to Papertrail.

remote_syslog can send both app and system log files. It doesn't modify the system-wide logging configuration at all and Papertrail does not need the IP addresses of your servers. Setup instructions.

C. Upgrade to syslog-ng or rsyslog.

These are better loggers by any measure. CentOS and Gentoo were the last distributions to switch.

Test: Basic IP Reachability

To see whether your system can reach Papertrail, run:

telnet logs.papertrailapp.com 514

You should see a connection established ("Connected to..") but will not see any output. Here's an example:

$ telnet logs.papertrailapp.com 514

Connected to logs.papertrailapp.com.
Escape character is '^]'.
^]
telnet> close

Note that this only tests TCP reachability to the default syslog port. It does not test UDP reachability.

Test: Hostname Format

The syslog packet includes a hostname field. Papertrail accepts syslog messages with hostnames that can exist in DNS (letters, numbers, periods, hyphens, and to be permissive, underscores).

Occasionally the operating system will have an invalid hostname like (none) or the syslog daemon will be transmitting a hostname value different from the operating system hostname.

OS hostname

Run:

hostname
hostname -f

Check the output to verify that the system hostname doesn't contain parentheses or other non-DNS characters.

Syslog hostname

Using syslog-ng? Look for the chain_hostnames option in syslog-ng.conf. If found, comment out that line, run /etc/init.d/syslog-ng restart, and retry the test.

This will ensure that syslog-ng does not transmit hostnameA@hostnameB as the hostname. (If you use chained hostnames, contact us or consider keep_hostname).

Test: Sending data

To see whether your logger(s) are actually sending packets to Papertrail, run:

sudo tcpdump -n -s 1500 -X port 11111

.. where 11111 is the port that the logger is sending to Papertrail on. While tcpdump is running, generate some log messages. You should see roughly one packet in the tcpdump output for each log message.

The output from tcpdump may be extremely verbose. To output it to a file, such as to send to Papertrail support, run:

sudo tcpdump -n -s 1500 -X port 11111 -w tcpdump.pcap

Test: Examine logger daemon activity

To see the actual behavior of the logger daemon (like rsyslog or remote_syslog), use strace. For example:

sudo strace -s 500 -tfp 22222

.. where 22222 is the process ID of the logger, as seen in the second column of ps auxw. While strace is running, generate some log messages. The output from strace may be extremely verbose. To output it to a file, such as to send to Papertrail support, run:

sudo strace -s 500 -tfp 22222 -o strace.log

If the strace program is not found, install your distribution's package with a command like sudo yum install strace or sudo apt-get install strace.

Test: System Logging Config

To generate a log message that uses your system syslogd config (/etc/rsyslog.conf, /etc/syslog-ng.conf, or another syslog config), run:

logger "test"

That will generate a test message containing "test" as a syslog message.

Note: if your system syslog is not setup properly (or cannot communicate with the remote syslog destination), this test message will also be affected by that problem. Try the instructions below to send a message directly to a host & port, without using the system's syslog.

Test: Remote Syslog

Install remote_syslog, create a placeholder log file, then tell remote_syslog to watch it and send new lines to a hostname/port provided by Papertrail, and not to daemonize. Example for <host>.papertrailapp.com:12345:

touch /tmp/testfile
remote_syslog -d <host>.papertrailapp.com -p 12345 -D /tmp/testfile

Then in another terminal or with remote_syslog backgrounded, append lines to the file:

echo "This is a test message" >> /tmp/testfile

Each message should appear on Papertrail (or your syslog destination) momentarily.

Stop remote_syslog with Ctrl-C when finished testing.

Suggestions

Test message arrived

The problem is probably with system or app server log configuration. Consider running syslogd in debug mode and in the foreground to look for error messages. Use: rsyslog -dn or syslog-ng -dF.

Test message did not arrive

The problem is probably with network reachability. If the system is behind a firewall, verify that firewall is not dropping log messages (UDP destined to the port above). If the system is behind NAT, verify that the public NAT IP matches the IP in Papertrail's system settings.

Firewall rules

To add an explicit firewall rule with iptables that allows outgoing traffic based on a destination port (recommended), run these 2 commands. Change 514 to to the destination port:

iptables -I OUTPUT 1 --protocol udp --dport 514 -j ACCEPT
iptables -I OUTPUT 1 --protocol tcp --dport 514 -j ACCEPT

To allow based on destination IP, add rules allowing traffic to 67.214.208.0/20 and 173.247.96.0/19:

iptables -I OUTPUT 1 -d 67.214.208.0/20 -j ACCEPT
iptables -I OUTPUT 1 -d 173.247.96.0/19 -j ACCEPT

.. then generate another test message. If this works, make sure to save these firewall rules for the next reboot.

SELinux

If your system enforces SELinux policies and the destination port is not 514, you may need to add the port to the set of ports which syslogd can open network connections to.

To confirm that SELinux is the cause, execute strace -tt -s 500 -fp PID -o strace.log (replace PID with the pid of your logging software), and then grep for EACCESS:

grep EACCESS strace.log

Output similar to the following will indicate that a policy change is required:

connect(6, {sa_family=AF_INET, sin_port=htons(XXXX), sin_addr=inet_addr("67.214.X.X")}, 16) = -1 EACCES (Permission denied)

To permit access, on RHEL6:

yum install policycoreutils-python
semanage port -a -t syslogd_port_t -p tcp 12345

To see current port settings:

semanage port --list

Message arrived but was attributed to incorrect system

The problem is probably with the name which the sender is transmitting to Papertrail (and which Papertrail uses to identify to sender). Contact us and we'll make specific suggestions for your hosting environment.

Allowing Outgoing Traffic on Juniper/Netscreen ScreenOS based Firewalls

To allow log traffic to pass through the firewall, configure it via the Web interface as follows:

Creating a Custom Service

Applying the Service