Troubleshooting remote syslog reachability

After configuring a system to log to Papertrail, if logs aren’t appearing, or aren’t appearing as expected, these checks help verify end-to-end reachability from your system to Papertrail (with few or no system changes). This can identify whether a problem is with the system/logger configuration, network reachability, or Papertrail configuration.

We’re here to help and we love this stuff, so don’t feel like you need to go through this process alone. Chat or email us here.

Contents

Tests

Configuration and limits

Tests

Basic IP Reachability

To see whether your system can reach Papertrail, run:

$ telnet logs.papertrailapp.com 514

You should see a connection established (“Connected to…”) but won’t see any output. Here’s an example:

$ telnet logs.papertrailapp.com 514

Connected to logs.papertrailapp.com.
Escape character is '^]'.
^]
telnet> close

Note that this only tests TCP reachability to the default syslog port. It doesn’t test UDP reachability. Replace logs and 514 with the hostname and port number for the account destination to test TCP connectivity to alternate ports.

Hostname Format

The syslog packet includes a hostname field. Papertrail accepts syslog messages with hostnames that can exist in DNS (letters, numbers, periods, hyphens, and to be permissive, underscores).

Occasionally, the operating system will have an invalid hostname like (none), or the syslog daemon will be transmitting a hostname value different from the operating system hostname.

OS hostname

Run:

$ hostname
$ hostname -f

Check the output to verify that the system hostname doesn’t contain parentheses or other non-DNS characters.

Syslog hostname

Using syslog-ng? Look for the chain_hostnames option in syslog-ng.conf. If found, comment out that line, run /etc/init.d/syslog-ng restart, and retry the test.

This will ensure that syslog-ng does not transmit hostnameA@hostnameB as the hostname. (If you use chained hostnames, contact us or consider keep_hostname).

Sending data

To see whether your logger(s) are actually sending packets to Papertrail, run:

$ sudo tcpdump -n -s 1500 -X port XXXXX

…where XXXXX is the port that the logger is sending to Papertrail on. While tcpdump is running, generate some log messages. You should see roughly one packet in the tcpdump output for each log message.

The output from tcpdump may be extremely verbose. To output it to a file, such as to send to Papertrail support, run:

$ sudo tcpdump -n -s 1500 -X port 11111 -w tcpdump.pcap

Examine logger daemon activity

To see the actual behavior of the logger daemon (like rsyslog or remote_syslog2), use strace. For example:

$ sudo strace -s 500 -tfp 22222

…where 22222 is the process ID of the logger, as seen in the second column of ps auxw. While strace is running, generate some log messages. The output from strace may be extremely verbose. To output it to a file, such as to send to Papertrail support, run:

$ sudo strace -s 500 -tfp 22222 -o strace.log

If the strace program is not found, install your distribution’s package with a command like sudo yum install strace or sudo apt-get install strace.

System Logging Config

To generate a log message that uses your system syslogd config (/etc/rsyslog.conf, /etc/syslog-ng.conf, or another syslog config), run:

$ logger "test"

to generate a message containing “test” as a syslog message.

If your system syslog is not set up properly (or cannot communicate with the remote syslog destination), this test message will also be affected by that problem. Try the instructions below to send a message directly to a host & port, without using the system’s syslog.

Use remote_syslog2

Install remote_syslog2, create a placeholder log file, then tell remote_syslog2 to watch it and send new lines to a hostname/port provided by Papertrail, and not to daemonize:

$ touch /tmp/testfile
$ remote_syslog -d logsN.papertrailapp.com -p XXXXX -D /tmp/testfile

Replace logsN and XXXXX with the host and port provided on log destinations. Then in another terminal or with remote_syslog2 backgrounded, append lines to the file:

$ echo "This is a test message" >> /tmp/testfile

Each message should appear in Papertrail momentarily.

Stop remote_syslog2 with Ctrl-C when finished testing.

Interpreting tests

The tests above are intended to help guide to the likely cause. Choose the test result below.

Test message arrived

The problem is probably with system or app server log configuration. Start here.

For example, consider running the log sender in debug mode and in the foreground to look for error messages, such as rsyslogd -dn or syslog-ng -dF.

Message arrived but was attributed to incorrect system

The problem is probably with the name which the sender is transmitting to Papertrail (and which Papertrail uses to identify to sender). Contact us and we’ll make specific suggestions for your hosting environment.

Test message did not arrive

There are a few possible causes:

  • The most likely is network reachability. If the system is behind a firewall, verify that the firewall policy permit log messages destined to the relevant port.
  • If the system is behind NAT and you provided Papertrail with a source IP, verify that the public NAT IP matches the IP in Papertrail’s system settings.
  • Ensure that “Yes, recognize logs from new systems” is enabled under the log destinations settings page, or that the system has been added using papertrail-add-system.

I’m not sure

Say hello and we’ll help. Please include anything you’ve already tried.

Message Length

UDP

Papertrail accepts UDP packets of up to 1024 bytes (1 KB) in total packet length, which is about 950 characters for the log message text. That’s a very long line - 5-10 lines of typical dense text.

Note that the limit is applied separately to each packet (log line). Almost all lines longer than 1 KB are spread across multiple lines anyway (for example, tracebacks). As a result, this limit is almost never encountered.

1024 bytes is a protocol requirement defined in RFC 3164, which nearly all senders enforce (either by dropping longer messages or splitting them into multiple conforming messages). Even without the RFC requirement, UDP messages would be constrained by the Internet MTU of 1500 bytes minus the UDP header.

TCP

Papertrail accepts messages over TCP that are up to 100,000 bytes (~100 KB) long. 100 KB was chosen as the outer edge of what we could imagine a use for and to ensure Papertrail can handle messages from default configurations of all common syslog daemons. Messages larger than 100KB will be dropped, and the connection temporarily closed.

Message Format

Papertrail accepts messages that are formatted according to either RFC 3164 or RFC 5424. Senders may use either or both. No configuration within Papertrail is necessary.

While RFC 5424 defines the hostname field to be up to 255 characters, Papertrail truncates the hostname value to 120 characters.

Papertrail also accepts many types of malformed messages, generally because the message formats are generated by commonly-used software and hardware.

Sending over TCP

By default, Papertrail accepts messages over TCP with TLS. Unencrypted TCP is also supported, but not enabled by default. To enable it, visit Destinations.

Firewall policies

To add an explicit firewall rule with iptables that allows outgoing traffic based on a destination port (recommended), run these 2 commands. Change 514 to to the destination port:

$ iptables -I OUTPUT 1 --protocol udp --dport 514 -j ACCEPT
$ iptables -I OUTPUT 1 --protocol tcp --dport 514 -j ACCEPT

To allow based on destination IP, add rules allowing traffic to 67.214.208.0/20, 173.247.96.0/19, and 169.46.82.160/27:

$ iptables -I OUTPUT 1 -d 67.214.208.0/20 -j ACCEPT
$ iptables -I OUTPUT 1 -d 173.247.96.0/19 -j ACCEPT
$ iptables -I OUTPUT 1 -d 169.46.82.160/27 -j ACCEPT

…then generate another test message. If this works, make sure to save these firewall rules for the next reboot.

When using IP subnet-based egress firewall rules, please sign up for Papertrail’s log destination changes mailing list. This is only necessary if your network configuration requires you to know about every new or deprecated log destination subnet. It’s not necessary if you use a current IP as your log destination only because the sender doesn’t support sending to a domain. Papertrail tracks inbound messages and will email account contacts before removing any destinations, so you’ll always hear from us if you need to change an existing IP.

SELinux

If your system enforces SELinux policies and the destination port is not 514, you may need to add the port to the set of ports which syslogd can open network connections to.

To confirm that SELinux is the cause, execute strace -tt -s 500 -fp PID -o strace.log (replace PID with the pid of your logging software), and then grep for EACCESS:

$ grep EACCESS strace.log

Output similar to the following will indicate that a policy change is required:

connect(6, {sa_family=AF_INET, sin_port=htons(XXXX), sin_addr=inet_addr("67.214.X.X")}, 16) = -1 EACCES (Permission denied)

To permit access, on RHEL6:

$ yum install policycoreutils-python
$ semanage port -a -t syslogd_port_t -p tcp 12345

To see current port settings:

$ semanage port --list

Juniper/Netscreen ScreenOS

To allow log traffic to pass through the firewall, configure it via the Web interface as follows:

Creating a Custom Service

  • Navigate to Policy > Policy Elements > Services > Custom
  • Set the service name to Papertrail
  • Set the Service Timeout to Use protocol default
  • Select either UDP or TCP depending on how you wish to log
  • Set the lowSource Port to 0 and the highSource Port to 65535
  • Set the low and highDestination Port to the port of the log destination that you wish to send log data to
  • Click Ok to save the changes

Applying the Service

  • Navigate to Policy > Policies
  • Select Trust from the From dropdown and Untrust from the To dropdown
  • Click New at the top right to create a new policy
  • Under Name specify Papertrail
  • Set the Source Address & Destination Address to Any from the drop down
  • Click the Multiple button and select the Papertrail service from the list
  • Set the Action to Permit
  • Click Ok to save the policy

Idle timeouts

Papertrail automatically closes idle TCP connections after 15 minutes of inactivity. Log senders should already handle this as part of a full implementation, since connections are routinely affected by Internet reachability changes. Senders can elect to immediately reconnect, or leave the connection closed until new log messages are available.

In addition to Papertrail’s timeout, Google Compute Engine and Microsoft Azure also have their own.

Rate limits

The following rate limits apply for log messages transmitted to Papertrail:

For UDP:

  • 3,000 messages per second per source IP
  • 10,000 messages per second to a log destination (port)

For TCP:

  • 10 new connections per second per source IP to a log destination (port)

For more information on why they were introduced, see Introducing Syslog Rate Limits.

Hitting these limits? Let us know. Please include the logger name and associated configuration file.

GNU syslogd

Test

Until recently, a few distributions (CentOS, Gentoo) provided GNU syslogd rather than syslog-ng or rsyslog. GNU syslogd only supports logging to the default syslog port (514), not user-selected destination ports. To see whether you run GNU syslogd, check:

  • Did you edit /etc/syslog.conf (rather than another filename)?
  • Run: syslogd -v. Does it say syslogd 1.4.1 (or another version)?

If so, this may be the problem.

Workarounds

Papertrail has a few workarounds for GNU syslogd. Choose one:

Tell Papertrail your system’s IP address

  1. Visit Add Systems and select “I’d like to aggregate system/OS logs from something else…”.
  2. Under Less common setup methods below the search area, click My syslog daemon only sends to port 514.
  3. Provide Papertrail with this device’s Internet-facing public (external NAT) IP.

Use the standalone remote_syslog2 log collector

remote_syslog2 can send both app and system log files. It doesn’t modify the system-wide logging configuration at all and Papertrail does not need the IP addresses of your servers. Setup instructions.

Upgrade to syslog-ng or rsyslog

These are better loggers by any measure. CentOS and Gentoo were the last distributions to switch.