Alerts

Here's how to send matching log messages to email, services like Campfire or Librato Metrics, and your own custom HTTP webhooks.

Introduction

Papertrail can notify external services when new log messages match important searches. These notifications can happen every minute (like for a monitoring system), every hour, or every day, and only occur in periods when Papertrail receives new matches for a given search.

For background, see the search alert announcement blog post.

Create alert

To create an alert, save a search, then attach an alert. Here's how.

1. Save a search. In Events, search for the logs which Papertrail should alert on. Once the matches are the set which are important for this alert, click Save Search.

save_search_example.jpg

2. Attach an alert. Give the new saved search a name, and click Save & Setup an Alert:

save_and_setup_an_alert-1.jpg

Existing search: to add an alert to an existing saved search, visit the Dashboard, mouseover the saved search, and an Edit link will appear. Click it, then click the Manage Alerts tab.

Configuration

After following the instructions above, the last step is to choose the alert(s) and configure them. The Manage Alerts tab looks like this:

manage_alerts-1.jpg

Supported services

Papertrail can notify:

  • Boundary: annotate a Boundary network traffic graph with the log message. More.
  • Campfire: send a chat message to a Campfire room. It contains the logs and a link. More.
  • Emails: send an email to a set of addresses of your choosing.
  • GeckoBoard: update a custom "number" widget with the count of matches. More.
  • HipChat: send a chat message to a HipChat room. It contains the logs and a link. More.
  • Librato Metrics: graph the number of occurrences over time. More.
  • PagerDuty: invoke an alert escalation policy, such as to generate text messages. More.
  • StatHat: graph the number of occurrences over time. More.
  • Webhook: notify a HTTP URL of your choosing. See Web hooks.

Papertrail also supports a handful of non-alert Integrations.

Don't see one you need? Just ask or implement.

Schedule

Alerts are processed every minute, every hour at about the minute that the alert was created, and every day at about 5 AM in the timezone which the alert's messages will be timestamped to. Examples:

  • A daily alert with a timezone of Eastern Time will be invoked at about 5 AM Eastern Time.
  • An hourly alert created at 5:43 will be invoked at about 6:43, 7:53, and so forth.

NOTE: Alerts are not guaranteed to start and end on the same second. The impact of this is that 2 alerts with the same interval, that are associated with the same saved search, may not be triggered at the same time.

For example, say that one matching log line arrives at 09:01:10. A per minute alert that considered log lines from 09:00:09 - 09:01:09, wouldn't find any matches. Another alert that had a start time of 1 second later would fire as expected.

Minimum threshold

Optionally, choose how many matching events must occur in the time interval chosen. For example, if a search alert runs every 10 minutes and should only be invoked when 5 or more events occur during that window, enter 5. The default and most common value is 1, which means to invoke the alert any time at least one matching message has occurred.

Maximum

For each alert invocation, Papertrail will truncate matching log messages to at most:

  • 25,000 log messages, and
  • 10 megabytes of JSON-encoded log messages (including related fields)

Custom services

To create your own alert service or extend the set of services that Papertrail supports, visit Web hooks.