Web hooks

React to log events: when Papertrail receives new log events that match one of your saved searches, it can notify services like Campfire, Librato, or a HTTP URL that you provide (web hooks).

Sample app

Here’s the source to a working example web hook handler on GitHub. This tiny Sinatra app accepts a web hook request from Papertrail containing log matches. It then submits them to a remote Web service, Prowl.

The papertrail-services handlers may also be useful.

Here’s a tiny client which POSTs example log data on GitHub.

Web hooks operated by Papertrail

Papertrail operates a set of web hooks called papertrail-services for common services like Campfire and email. These are available in Papertrail’s Web interface. From the Dashboard, click the pencil icon on an existing saved search, then scroll down to the Create an Alert section:

manage_alerts.png

Papertrail hosts these alert processors.

They are also available as a standalone open-source Sinatra Web service, which you can run yourself, fork and modify, and even contribute new services which you want Papertrail to run on your behalf.

The papertrail-services README has more.

You can also write your own standalone web hook handler without using papertrail-services. The rest of this page covers how to do that.

Example use

You have a monitoring system running in your datacenter or on another hosted service. New log events with the string segfault are an important failure case, and they should be sent to the monitoring system so it can page administrators.

Papertrail will notify your monitoring system every minute that new matches are found, and will also provide the contents of the new log matches.

Papertrail can notify every hour or day for less-urgent message types, as for generating summaries.

Setup

  1. In Events, search for the relevant query (in the example use above, “segfault”)
  2. Click “Save Search” and give the search a name.
  3. From the Dashboard, edit the search. Define your webhook URL and callback frequency (minute, hour, or day)

If the webhook only needs a count of matching logs, not all raw logs, enable the “Send only counts” option. See Count-only webhooks.

Callback

The callback will be a POST request in the format used by the github-services suite of webhook handlers. The POST body will contain a single parameter called payload, which will contain a JSON hash.

The important key in this hash is:

Example

For example, here is a POST body containing 2 events:

payload = {
  "events": [
    {
      "id": 7711561783320576,
      "received_at": "2011-05-18T20:30:02-07:00",
      "display_received_at": "May 18 20:30:02",
      "source_ip": "208.75.57.121",
      "source_name": "abc"
      "source_id": 2,
      "hostname": "abc",
      "program": "CROND",
      "severity": "Info",
      "facility": "Cron",
      "message": "message body",
    },
    {
      "id": 7711562567655424,
      "received_at": "2011-05-18T20:30:02-07:00",
      "display_received_at": "May 18 20:30:02",
      "source_ip": "208.75.57.120",
      "source_name": "server1"
      "source_id": 19,
      "hostname": "def",
      "program": "CROND",
      "severity": "Info",
      "facility": "Cron",
      "message": "A short event",
    }
  ],
  "saved_search": {
    "id": 42,
    "name": "Important stuff",
    "query": "cron OR server1",
    "html_edit_url": "https://papertrailapp.com/searches/42/edit",
    "html_search_url": "https://papertrailapp.com/searches/42"
  },
  "max_id": 7711582041804800,
  "min_id": 7711561783320576,
  "frequency": "1 minute"
}

Note: For readability, the example above is not URL-encoded. See Encoding below. Also, whitespace and line breaks have been added.

The following hash keys are defined for each log event:

HTTP and HTTPS URLs are supported.

To learn more about the meaning of each column, see response field descriptions in HTTP API.

Encoding

The payload JSON hash is encoded with application/x-www-form-urlencoded, as is standard for all POST requests. Apps rarely need to consider this because almost all Web frameworks automatically URL-decode the POST contents. In those cases, the app only needs to decode JSON.

Here’s a sample test client to simulate a request from Papertrail.

Here’s a very minimal example of how the request is generated. This uses the Ruby irb console and the payload event only contains a hostname, but it shows the format of the URL-encoded POST:

>> payload = { :events => [ { :hostname => 'abc' } ] }
=> {:events=>[{:hostname=>"abc"}]}

>> puts payload.to_json
{"events":[{"hostname":"abc"}]}

>> puts URI.escape(payload.to_json)
%7B%22events%22:[%7B%22hostname%22:%22abc%22%7D]%7D

Parsing

In Ruby with the Ruby on Rails framework, this will turn the callback POST params into an Ruby hash called payload:

payload = Yajl::Parser.parse(params[:payload])

Implementation considerations

Up to 25,000 events are included in a single callback. That was chosen as a compromise based on:

The attribute reached_record_limit is set in the top-level hash when the 25,000 limit is reached. If you need more, just ask and we’ll try to make it work.

Papertrail uses 64-bit event IDs, which JSON does not support, so the id column is a string. The other columns are consistently set to the type you would expect. No columns should be null except for program (when none is defined by the message). An empty message body is a blank string (‘’).

Testing

To receive a web hook containing a fixed set of log data, use the test client mentioned above.

To feed real log data from your Papertrail account into a local development environment, use a service such as:

Alternatively, you can host your code in the cloud. A list of providers can be found here.

Count-only webhooks

Most webhooks act on the message body or other parts of the log message. However, for webhooks which only need the number of matching events, or do not need any information at all (beyond that the alert fired), enable the “Count-only webhooks” option.

Since a webhook invocation may cover thousands of matching messages, sending only counts can turn a 5 MB JSON payload into only a few KB.

When this option is enabled, the canonical example is modified. No events hash key is included. Instead, a counts hash key contains an array of hashes, one hash per sending system seen during the period.

Each sender hash has the sending system’s name and ID (more). It also has a timeseries key containing a count of messages received from that sender at each Unixtime timestamp. The key is a Unixtime timestamp (as a JSON integer). The value is the number of matching messages generated by the sender during that 1 second interval.

Here is an example payload for a count-only webhook invocation covering 16 total events. For example, the sender arthur sent 5 messages at timestamp 1453248895.

Other than the new counts and omitted events keys, the payload is identical to the canonical example:

 payload = {
   "counts": [
     {
       "source_name": "arthur",
       "source_id": 4
       "timeseries": {
         1453248892: 2,
         1453248895: 5
         1453248948: 1
         1453248905: 5
       }
     },
     {
       "source_name": "ford",
       "source_id": 3
       "timeseries": {
         1453248927: 3
       }
     },
   ],
   "saved_search": {
     "id": 42,
     "name": "Important stuff",
     "query": "cron OR server1",
     "html_edit_url": "https://papertrailapp.com/searches/42/edit",
     "html_search_url": "https://papertrailapp.com/searches/42"
   },
   "max_id": 7711582041804800,
   "min_id": 7711561783320576,
   "frequency": "1 minute"
 }