Papertrail Knowledge Base

Alerts

Here's how to receive certain log messages to email or send them to services like Campfire, Slack, Librato Metrics, and your own custom HTTP webhooks.

Introduction

Papertrail can notify external services when new log messages match important searches. These notifications can happen every minute (like for a monitoring system), every hour, or every day, and only occur in periods when Papertrail receives new matches for a given search.

For background, see the search alert announcement blog post.

Create alert

To create an alert, save a search, then attach an alert. Here's how.

1. Save a search. In Events, search for the logs which Papertrail should alert on. Once the matches are the set which are important for this alert, click Save Search.

save_search_example.jpg

2. Attach an alert. Give the new saved search a name, and click Save & Setup an Alert:

save_and_setup_an_alert-1.jpg

Existing search: to add an alert to an existing saved search, visit the Dashboard, click the pencil icon on the saved search, then scroll down to Create an Alert.

Configuration

After following the instructions above, the last step is to choose the alert(s) and configure them. The Create an Alert section looks like this:

manage_alerts.png

Supported services

Papertrail can notify:

In addition, Papertrail has non-alert integrations with services like Honeybadger, New Relic, OpsGenie, and an OS X Dashboard widget. Expand the "Integration" sidebar menu section for additional integrations.

Don't see one you need? Just ask or implement.

Schedule

Alerts are processed every minute, every hour at about the minute that the alert was created, and every day at about 5 AM in the timezone which the alert's messages will be timestamped to. Examples:

NOTE: Alerts are not guaranteed to start and end on the same second. The impact of this is that 2 alerts with the same interval, that are associated with the same saved search, may not be triggered at the same time.

For example, say that one matching log line arrives at 09:01:10. A per minute alert that considered log lines from 09:00:09 - 09:01:09, wouldn't find any matches. Another alert that had a start time of 1 second later would fire as expected.

Daily alert at a specific time

Daily alerts fire at approximately 5 AM in the timezone associated with the alert. The latter can be used to change when the alert executes in local time.

For example, if a daily alert needs to run at 8 AM Pacific standard time (GMT-8), it could be implemented by setting the alert's timezone to Samoa (GMT-11).

Note that timestamps inside alert notifications will also reflect the alert's timezone.

Minimum threshold

Optionally, choose how many matching events must occur in the time interval chosen. For example, if a search alert runs every 10 minutes and should only be invoked when 5 or more events occur during that window, enter 5. The default and most common value is 1, which means to invoke the alert any time at least one matching message has occurred.

Log velocity notifications

This can also serve as finer-grained notification when log velocity changes significantly. To use alerts as fine-grained velocity notifications, create a search which matches all logs (such as " "), then an alert with a relatively high minimum threshold. For example, a 1-minute alert interval could have a minimum of 30,000 in a minute, or an average of 500 per second.

Note that thresholds higher than 120,000 per minute (that is, 5,000/second) may yield less predictable alert invocations due to variations in Papertrail's processing rate. The alerts will likely serve the purpose as velocity notifications, but the specific rate should not be considered authoritative.

Maximum

For each alert invocation, Papertrail will truncate matching log messages to at most:

Custom services

To create your own alert service or extend the set of services that Papertrail supports, visit Web hooks.

Inactivity

To generate an alert on inactivity, such as if a cron job or background worker fails to run, combine Papertrail with a silence-detection service like Dead Man's Snitch.

After the system is logging output to Papertrail, then:

  1. Save a search which matches at least one log message generated by your periodic job.

  2. Define a webhook alert. For the webhook URL, give Papertrail the URL provided by DMS.

  3. Choose a Papertrail alert frequency longer than your intended monitoring frequency. For example, if a cron job is meant to run every 30 minutes, choose "Every hour" for the Papertrail alert frequency. Choose a similar desired check-in frequency on DMS.

As long as the search terms appear in one or more log lines, Papertrail will update the silence-detection service. If the search has no matches in a given alert frequency, Papertrail will not hit the URL and the silence-detection service will notify you.