Configuring centralized logging from Python apps

Papertrail can accept logs from any Python app, including Django.

For accounts created via the Papertrail website, see methods A & B. For Heroku add-on accounts, see the Heroku section below.

Method A: Use log file

Configure your app to log to a file as usual, then use remote_syslog2 to watch that file. Setup instructions.

Method B: Use Python’s SysLogHandler

Python can also send log messages directly to Papertrail with Python’s SysLogHandler logging handler.

Older minor versions of Python 2.7 and 3.2 may be subject to this bug. The fix is present in current 2.7 and 3.2 as well as all versions of 3.3 and later.

Configuration

Example: Django

Building on this example, add a new handler:

'handlers': {
	'SysLog': {
		'level': 'DEBUG',
		'class': 'logging.handlers.SysLogHandler',
		'formatter': 'simple',
		'address': ('<host>.papertrailapp.com', 11111)
	},
	...
}

Change the address argument to match your Papertrail log destination.

To set the sender and program name, modify the simple formatter like so:

'formatters': {
	'simple': {
		'format': '%(asctime)s SENDER_NAME PROGRAM_NAME: %(message)s',
        'datefmt': '%Y-%m-%dT%H:%M:%S',
	},
	...
}

Then add the new logging handler to at least one of the loggers in the config, for example:

'loggers': {
	'django': {
		'handlers': ['file', 'SysLog'],
		'level': 'INFO',
		'propagate': True,
	},
	...
}

Example: Generic Python app

import logging
import socket
from logging.handlers import SysLogHandler

class ContextFilter(logging.Filter):
  hostname = socket.gethostname()

  def filter(self, record):
    record.hostname = ContextFilter.hostname
    return True

logger = logging.getLogger()
logger.setLevel(logging.INFO)

f = ContextFilter()
logger.addFilter(f)

syslog = SysLogHandler(address=('<host>.papertrailapp.com', 11111))
formatter = logging.Formatter('%(asctime)s %(hostname)s YOUR_APP: %(message)s', datefmt='%b %d %H:%M:%S')

syslog.setFormatter(formatter)
logger.addHandler(syslog)

logger.info("This is a message")

Change the address argument to match your Papertrail account.

Example: Format specifiers

These format specifiers have not been extensively tested but may be helpful as starting points, especially combined with the logging format specifiers.

A valid syslog message looks like this:

<22>Jan 2 23:34:45 hostname app_name[PID]: message

The timestamp can also be in other formats as shown in the code snippet above.

Here’s an example format specifier for having Python’s SysLogHandler generate syslog or syslog-like messages:

logging.Formatter('%(asctime)s %(hostname)s APP: %(message)s', datefmt='%b %d %H:%M:%S')

The system hostname can be obtained with socket.gethostname(), by honoring the HOSTNAME environment variable with os.getenv('HOSTNAME'), or hardcoded as a static string literal. SysLogHandler automatically prefixes the message with a priority code, which is <22> in the example syslog message shown above.

Here’s an alternative example that may be useful, depending on your environment:

%(name)s[%(process)d]: %(levelname)s %(message)s

The DatagramHandler may be used to similar effect.

Example: Multiline log Messages

Syslog is a simple protocol that transports single lines of text. As a result, no encoding is provided for newlines. The message has to be split and separate packets have to go out over the wire.

For example, to log an exception, you’d use something like:

tb = ""
try:
  a = 1/0
except:
  tb = traceback.format_exc()

lines = tb.split('\n')
for l in lines:
  logger.info(l)

Logging via the Heroku add-on

Rather than logging directly to syslog, send everything to stdout (the console) and Heroku will forward it over to Papertrail.

The following handler will allow logging to the console:

import sys
LOGGING = {
    'handlers': {
        'console':{
            'level':'INFO',
            'class':'logging.StreamHandler',
            'strm': sys.stdout
        },
        ...
    }
}

Replace strm with stream if you’re using python 2.7.