Advanced Unix logging tips

Occasionally Asked Questions.

Note: If you’re just getting started and not already familiar with logging, start on Setup Systems, not this document.

Contents

Common tasks

rsyslog

FreeBSD



Common tasks

Override the hostname sent by a logger

Papertrail honors the hostname sent by your log sender, such as rsyslog or remote_syslog2. By default this is the system hostname. You can always rename a sender within Papertrail’s Web interface by visiting the Dashboard, clicking the All Systems link, then clicking the Edit Settings link for the sender. Changing the visible name does not affect event attribution.

However, sometimes a system may have a hostname like ip-23-56-78-12 or domU-49835 that isn’t very meaningful. You can explicitly configure the log sender to use a hostname of your choosing, and its messages will appear in Papertrail under that system name. Here’s how.

rsyslog

To change the hostname rsyslog sends, add the following directive as the very first line in /etc/rsyslog.conf before any modules are loaded:

$LocalHostName yourhostname

Alternatively, to have rsyslog send with the fully-qualified domain name (FQDN, such as system1.example.com) instead of simply the hostname (system1), use the directive:

$PreserveFQDN on

This is rarely needed. We recommend using the hostname (without the domain name) unless you have identically-named systems.

An alternative way to set it (that allows you to send different logs as different hostnames) is by setting a custom template:

$template MyTemplate, "<%pri%> %timestamp% MySpoofedHostName %syslogtag% %msg%\n"
$ActionForwardDefaultTemplate MyTemplate

syslog-ng

Add the following above the destination stanza that sends to Papertrail:

template t_papertrail {
  template("<$PRI>$DATE yourhostname $PROGRAM: $MSG\n");
};

.. where yourhostname is the hostname or IP that you want Papertrail to recognize this system as.

In the default template format, syslog-ng uses $HOST (the system hostname). This is effectively replacing a passthrough option with a static string of your choosing.

Apply it by adding the template directive to the tcp or udp options inside the destination stanza:

destination d_papertrail {
  udp("<host>.papertrailapp.com" port(XXXXX) template(t_papertrail) );
};

remote_syslog2

To change the hostname remote_syslog2 sends from, pass the --hostname option on the command line:

remote_syslog --hostname=yourhostname

or use the configuration directive in the log_files.yml:

hostname: yourhostname

Import an existing log file to Papertrail

This is almost never needed and very rarely leads to a great result. Two cases where it may help:

There are two relatively easy ways to import existing log messages to Papertrail. Timestamps in the log file are not honored but the message contents will be searchable.

remote_syslog2

Run remote_syslog2 and point it at a temporary file, then use cat to append the existing log file contents to that temp file. Here’s an example where /tmp/placeholder is the temp file:

touch /tmp/placeholder
remote_syslog -D -d <host>.papertrailapp.com -p 11111 /tmp/placeholder
cat /home/httpd/logs/missing-file.log >> /tmp/placeholder

Then wait a minute or two so remote_syslog2 can send the newly-appended logs (depending on how large the file is). Once you see them in Papertrail, kill remote_syslog2.

syslog daemon

If your system is transmitting syslog (like from rsyslog or syslog-ng), you can “piggyback” on that and feed a text file into syslog. The Unix logger command accepts a filename and sends each line to the system syslog daemon. Here’s an example:

logger -f the-missing-file.log

Again, this is very rarely needed.

Aggregate output from a command or cron job

To capture output from a Unix command or a cron job, pipe it to the standalone Unix program logger. Anything piped to logger goes to the system syslog daemon and thus on to Papertrail (as long as syslog is sent to Papertrail).

Example:

echo "hi" | logger

You can also choose a tag for the logs, like a program name. For example:

uptime | logger -t uptime
uptime | logger -t bob-monitoring

Any Unix command can work this way, including cronjobs. Most cronjobs redirect standard error (file descriptor 2) to standard out (file descriptor 1). By default, cron emails it to you, but it can also be sent to logger:

some-command 2>&1 | logger

or tagged as cron:

some-command 2>&1 | logger -t cron

Setup a loghost which relays to Papertrail (or use an existing one)

Papertrail can receive logs directly from the originating systems and apps, or relayed through a local log host. Many syslog daemons can relay messages without modifying the syslog header. Because the relay does not modify the header, Papertrail will automatically attribute the message to the original sender, not the relay.

Here’s how to use rsyslog as a local aggregator that relays to Papertrail.

Installation

  1. Install a Linux distribution such as CentOS. This can be a tiny (256 MB) virtual machine on minimal hardware. See CentOS downloads. If prompted, choose to install without graphic (X Window System) support.
  2. Once booted, login as root and confirm that /etc/rsyslog.conf exists by running: ls -la /etc/rsyslog.conf
  3. Edit /etc/rsyslog.conf and uncomment these 4 lines:
$ModLoad imudp.so
$UDPServerRun 514
$ModLoad imtcp.so
$InputTCPServerRun 514
  1. Allow systems to log to your system. Note: You may want to restrict this to only expected source IPs. Adjust according to your network topology.
sudo iptables -A INPUT -m state –state NEW -m udp -p udp –dport 514 -j ACCEPT
sudo iptables -A INPUT -m state –state NEW -m tcp -p tcp –dport 514 -j ACCEPT
sudo service iptables save
  1. Configure other systems and apps to log to this new system. Feel free to use Papertrail’s instructions, but instead of configuring them to log directly to Papertrail (such as <host>.papertrailapp.com:1111), point them at this system (such as myloghost.example.com:514 or 192.168.1.42:514).

Finally, continue to Configuration below to enable Papertrail.

Configuration

  1. Configure rsyslog according to Papertrail’s standard instructions on Add Systems. The same configuration will relay not only logs from this system, but also logs received from other systems.
  2. Restart rsyslog: sudo service rsyslogd restart

As soon as a system generates a new message, it will be visible in Papertrail with the original system’s name.



rsyslog

Tweak queue options for connection failure

This section is irrelevant for those using the default settings, which log via UDP. There is no concept of receipt verification or connections.

If you have setup TCP with TLS, rsyslog is aware of whether it can connect to and transmit to Papertrail. If it can’t, it needs to know what to do with incoming syslog messages. This can be to drop them or to queue them in memory or in memory and on disk.

While a full discussion of rsyslog queues is outside the scope of this document, here’s a sample configuration that queues up to 100,000 lines in memory and on disk, tries to reconnect to Papertrail in perpetuity, and times out new syslog() calls in 2ms (so that your apps do not block if the queue fills). This would go immediately before the *.* @@<host>.papertrailapp.com line in your rsyslog config:

$ActionResumeInterval 10
$ActionQueueSize 100000
$ActionQueueDiscardMark 97500
$ActionQueueHighWaterMark 80000
$ActionQueueType LinkedList
$ActionQueueFileName papertrailqueue
$ActionQueueCheckpointInterval 100
$ActionQueueMaxDiskSpace 2g
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on
$ActionQueueTimeoutEnqueue 2
$ActionQueueDiscardSeverity 0

Aggregate local log files with rsyslog instead of remote_syslog2?

The easiest way to aggregate local log files is with Papertrail’s tiny standalone remote_syslog2 daemon, which is documented here.

Papertrail also works great with rsyslog or syslog-ng collecting local log files. Most rsyslog configs use /etc/rsyslog.conf to load additional files from /etc/rsyslog.d/. If you have this directory (see ls /etc/rsyslog.d/), create /etc/rsyslog.d/95-papertrail.conf with the Papertrail config. The 95 means that the config will be loaded near the end. If the /etc/rsyslogd.d/ directory does not exist, append the config to /etc/rsyslog.conf instead.

Here’s an example annotated rsyslog.conf snippet which will transmit one or more log files in realtime:

# load module to read from local files
$ModLoad imfile

# use a non-default ruleset (keeps logs out of /var/log/)
$RuleSet papertrail
$InputFileBindRuleset papertrail

# for each local log file path, duplicate the 5 lines below and edit lines 1-3
$InputFileName /path/to/some.log
$InputFileTag somelog:
$InputFileStateFile papertrail-somelog
$InputFilePersistStateInterval 100 # update state file every 100 lines
$InputRunFileMonitor

# destination (see https://papertrailapp.com/systems/setup)
*.* @<host>.papertrailapp.com:1111
# for clarity, explicitly discard everything (this is typically not necessary)
*.* ~

# all done. change to default ruleset (RSYSLOG_DefaultRuleset) for any following config
$InputFileBindRuleset RSYSLOG_DefaultRuleset
$RuleSet RSYSLOG_DefaultRuleset

Finally, inform rsyslog of the change with sudo /etc/init.d/rsyslog restart. We also recommend explicitly defining a WorkDirectory directive; see below.

State files (WorkDirectory)

rsyslog uses state files to track which parts of a log file have already been transmitted. By default, these state files are written in a directory chosen by the WorkDirectory option. rsyslog needs permission to write to that directory. To see the directory, run:

grep -r WorkDirectory /etc/*rsyslog*

Many distributions have rsyslog drop to a non-root user. For example, Ubuntu uses the syslog user. See which user rsyslog runs as with ps auxw | grep rsyslog | grep -v grep. Confirm that the WorkDirectory is writable by that user.

If you don’t find any explicit configuration lines, rsyslog may attempt to use /rsyslog/work/ or another path chosen by the rsyslog package maintainer. This directory probably doesn’t exist and wouldn’t be the best place for spool files if it did. To change to another directory, add this configuration option:

$WorkDirectory /var/run

RuleSets

The configuration above uses RuleSets to explicitly keep messages from local files out of the OS syslog. If your rsyslog version doesn’t support the RuleSet command, remove both RuleSet commands from the config stanza above and ensure that the Papertrail config stanza is last in the config file.

rsyslog configs are processed sequentially, so OS syslog messages will have already been written by the time that these files are read in.

Some messages from rsyslog are attributed to an IP address (often “Last message repeated…” messages).

This is a bug in rsyslog which was fixed on June 11, 2012. Either upgrade rsyslog, ignore the relatively few Last message repeated n times messages which are attributed to IPs, or set $RepeatedMsgReduction off to disable these messages entirely (at the expense of potentially higher output).

A Xen/EC2 hostname is domU-... when a new system boots (then changes to the hostname I’ve configured)

The rsyslogd daemon obtains the system hostname at the time it starts. Often it is started prior to the OS setting the hostname which you have configured. This can lead to the initial boot log messages appearing under the machine-chosen hostname like domU-11-22-33-44-55-66.

A few minutes or a few hours later, the log hostname changes to the one you configured. The change often happens the following evening when rsyslogd is notified of a log rotation and detects the new hostname.

There are a few simple ways to handle this:

Tell rsyslog to drop messages from certain syslog facilities?

To exclude all messages from a specific facility, change *.* to include a statement that drops messages in that facility. For example, to drop messages in the facility called cron, use cron.none. This will tell rsyslog to send all messages except cron:

*.*;cron.none             @<host>.papertrailapp.com:1111

More: facilities; Google query

rsyslog not transmitting anything or logging to disk

If new messages are not appearing in the local log files, and no messages are being transmitted to Papertrail, edit /etc/logrotate.d/rsyslog and change:

postrotate
        reload rsyslog >/dev/null 2>&1 || true
endscript

To:

create 640 syslog adm
postrotate
        /sbin/restart rsyslog >/dev/null 2>&1 || true
endscript

This will restart the rsyslog daemon rather than reloading it, and ensure that new log files are created with the correct permissions and ownership.

Solution originally posted on Launchpad

ANSI color escape sequences appear as strings, not colors

By default, rsyslog escapes ANSI color codes prior to sending them to Papertrail. If your logs include ANSI color and traverse rsyslog, add this configuration directive near the top of rsyslog.conf:

$EscapeControlCharactersOnReceive off

Restart rsyslog with sudo service rsyslog restart. Future log messages should be transmitted with actual escape sequences and thus be colorized by Papertrail.



FreeBSD

syslogd is not transmitting anything

FreeBSD often is configured with syslogd set to both ignore incoming syslog messages and silently refuse to sent outgoing ones. To see whether FreeBSD is configured this way, run:

ps auxw | grep syslog | grep -v grep

If you see a line with the argument -ss (two s letters), then syslogd will neither accept nor transmit log messages. For example:

root  50908   0.0  0.0  12184   1400  ??  Ss   12:48PM     0:00.01 /usr/sbin/syslogd -ss

To change this, edit /etc/rc.conf and remove one of the s letters so that it is started with syslogd -s (one s), then restart syslogd.