HTTP API

Your apps can make HTTP requests to Papertrail to programmatically search for events and manage groups, saved searches, systems, and log destinations.

Usage

The REST API is entirely optional and not necessary for typical use of Papertrail. Example uses:

  • search or tail logs for post-processing (using the same API as papertrail-cli) (also see webhooks)
  • create a saved search, like for a string provided by a user of your app (using the same API as paper_wrap) (also see linking to searches)
  • obtain settings for a given system, like for automated configuration (also see Configuration)
  • create groups or update group membership (using the same API as paper_wrap) (also see group wildcards)
  • register or remove users, like to synchronize with an employee database

Overview

Authentication

All API requests are authenticated using either:

  • An API token in the X-Papertrail-Token HTTP header (recommended). Find your token in Profile
  • HTTP Basic Authentication with a papertrailapp.com username and password

URL

All Papertrail API requests are rooted at https://papertrailapp.com/api/v1/.

Example URL: https://papertrailapp.com/api/v1/systems.json.

Examples

These examples use curl, a command-line HTTP client.

Authenticate with an API token and list all systems:

curl -v -H "X-Papertrail-Token: abc123" https://papertrailapp.com/api/v1/systems.json

Authenticate with an API token and save a search named Important stuff:

curl -v -H "X-Papertrail-Token: abc123" -X POST --data 'search[name]=Important+stuff&search[query]=very+important' https://papertrailapp.com/api/v1/searches.json

Authenticate with an API token and search events for Critical error as a quoted phrase:

curl -v -H "X-Papertrail-Token: abc123" "https://papertrailapp.com/api/v1/events/search.json?q='Critical error'"

Authenticate with a username and password and update the name of group ID 12 to Customer servers:

curl -v -u "my@email.com":"s3kr3t" -X PUT --data 'group[name]=Customer+servers' https://papertrailapp.com/api/v1/groups/12.json

Authenticate with a username and password and add system ID 4321 to group ID 12:

curl -v -u "my@email.com":"s3kr3t" -X POST --data 'group_id=12' https://papertrailapp.com/api/v1/systems/4321/join.json

Format

JSON

Either:

  • Append .json to the request path, such as: /api/v1/systems.json or /api/v1/groups/42.json
  • Include format=json in query parameters

Responses containing multiple items are arrays. Responses about a single item are hashes. No root node is present.

Requests and Responses

Request data uses URL-encoded query parameters. Unless otherwise noted, attributes are part of a hash named after the item type, such as:

system[name]=New%20System&system[ip_address]=1.2.3.4

See curl examples above or response examples below.

Update requests may contain all attributes or only the attribute(s) which are to be modified. Absent attributes will be left unmodified.

Errors

Failed requests return 400 Bad Request. Authentication failures return 403 Unauthorized.

For failed requests, a JSON hash is returned containing a single key called message. For example:

{"message":"Sad panda"}

Hypermedia

API responses include URLs for obtaining details about, or performing actions on, related items. For example, a client can query for all systems, then follow URLs in the API responses to:

  • obtain detail about a specific system, then
  • obtain details about a specific group it is a member of, then
  • perform an event search for that group

As a result, an API consumer will often only need to hardcode one URL.

Hypermedia links are contained in a hash key called _links with a hash per relation. For example:

"name":"Production servers",
"_links":{
  "self":{
    "href":"http://.."
  },
  "html":{
    "href":"http://.."
  }
}

When they apply to the item, these relations are used:

  • self: Canonical API URL for item. Consume this for item details.
  • html: Canonical HTML URL for item. Link humans to this.
  • search: API URL to search for events from or about this item (where applicable).

API Endpoints

Within https://papertrailapp.com/api/v1/:

Search Events

Item Operation Verb Path Path Example Required
Events Search GET
events/search
events/search.json



Manage Systems

Item Operation Verb Path Path Example Required
Systems List GET
systems
systems.json
System Show GET
systems/<id>
systems/42.json
System Register POST
systems
systems.json
Varies; see details
System Update PUT
systems/<id>
systems/42.json
System Unregister DELETE
systems/<id>
systems/42.json
System Join Group POST
systems/<id>/join
systems/42/join.json
group_id
System Leave Group POST
systems/<id>/leave
systems/42/leave.json
group_id



Manage Saved Searches

Item Operation Verb Path Path Example Required
Saved Searches List GET
searches
searches.json
Saved Search Show GET
searches/<id>
searches/42.json
Saved Search Create POST
searches
searches.json
name; query
Saved Search Update PUT
searches/<id>
searches/42.json
Saved Search Delete DELETE
searches/<id>
searches/42.json



Manage Groups

Item Operation Verb Path Path Example Required
Groups List GET
groups
groups.json
Group Show GET
groups/<id>
groups/42.json
Group Create POST
groups
groups.json
name
Group Update PUT
groups/<id>
groups/42.json
Group Delete DELETE
groups/<id>
groups/42.json



Manage Log Destinations

Item Operation Verb Path Path Example Required
Log Destinations List GET
destinations
destionations.json
Log Destination Show GET
destinations/<id>
destinations/42.json


Manage Users

Item Operation Verb Path Path Example Required
Users List GET
users
users.json
User Invite POST
invite
invite.json
email, read_only
User Delete DELETE
users/<id>
users/42.json


Retrieve account usage

Item Operation Verb Path Path Example Required
Accounts List GET
accounts
accounts.json



Examples: Provisioning API

Manage Systems

Papertrail automatically recognizes most systems so that explicit configuration is not needed. See Add Systems. The REST API handles unusual cases like:

  • automatically registering lots of systems which need explicit configuration (hosting company)
  • changing settings on many systems (synchronizing with an existing system metadata source)
  • identifying systems which have stopped sending logs

In the examples below, whitespace is added for clarity.

List

[ 
  {
    "name":"www5",
    "id":3248,
    "ip_address":"1.2.3.4",
    "hostname":null,
    "last_event_at":null,
    "syslog": {
      "hostname":"logs.papertrailapp.com",
      "port":514
    },
    "_links": { 
      "self": {
        "href":"https://papertrailapp.com/api/v1/systems/www5.json"
      },
      "search": {
        "href":"https://papertrailapp.com/api/v1/events/search.json?system_id=3248"
      },
      "html": {
        "href":"https://papertrailapp.com/systems/www5"
      }
    },
  },

  { 
    # Another system
  }
]

Notes:

  • last_event_at may be a timestamp (in the zone of the API token owner) or null
  • top-level hostname may be a string or null.

Show

See List. Only the hash for the requested system is returned.

Register

Systems can be registered in 2 ways:

Typical system

  • system[name]: display name of the system (Required)
  • system[hostname]: hostname to filter by (is often the same as the name)
  • destination_id: id of the destination port the system will log to (obtained from Manage Log Destinations)
  • destination_port: destination port number the system will log to

Note on destinations: Either the destination_id or destination_port must be specified.

Note on identification: Either the system[hostname] must be provided (typical and recommended) or the system will be created as the fallback "last resort" on the destination port provided.

Example:

system[name]=My%20Big%20Server&destination_id=42&hostname=bigserver

Logging to the standard syslog port

When the system's IP address does not change, this is an option.

  • system[name]: display name of the system (Required)
  • system[ip_address]: source IP address of the system (Required)
  • system[hostname]: filter events to only those from this syslog hostname (Optional)

Example:

system[name]=My%20Big%20Server&system[ip_address]=2.3.4.5&system[hostname]=bigserver

Optional attributes

Optional with both methods:

  • system[description]: freeform description (optional)
  • system[auto_delete]: whether to automatically delete sender when dormant and representing no searchable logs (optional; default: false)

Update

See Register.

Groups

List

[ 
  { 
    "name":"All Systems",
    "id":31,
    "system_wildcard":"*",
    "systems":[
      # see "Systems - List" for example array
     ],
    "_links": { 
      "self": {
        "href":"https://papertrailapp.com/api/v1/groups/31.json"
      },
      "search": {
        "href":"https://papertrailapp.com/api/v1/events/search.json?group_id=31"
      },
      "html": {
        "href":"https://papertrailapp.com/groups/31" 
      },
    },
  },

  { 
    # Another group
  } 
]

Notes:

  • All systems (whether static members or matching via wildcard) will be included in systems.
  • system_wildcard may be a string or null.

Show

See List. Only the hash for the requested group is returned.

Create

  • Reqired: group[name]
  • Optional: group[system_wildcard], group[system_ids]

group[name]=us-west%20zone&group[system_wildcard]=*west*&group[system_ids][]=31&group[system_ids][]=62

Update

See Create.

Manage Saved Searches

List

[
  {
    "name":"Login rejections",
    "id":1,
    "query":"\"access denied\" OR ssh",
    "group": { 
      "name":"All Systems",
      "id":31,
      "_links": {
        "self": {
          "href":"https://papertrailapp.com/api/v1/groups/31.json"
        }
      }
    },
    "_links": {
      "self": {
        "href":"https://papertrailapp.com/searches/1.json"},
      "search": {
        "href":"https://papertrailapp.com/api/v1/events/search.json?q=%22access+denied%22+OR+ssh"},
      "html": {
        "href":"https://papertrailapp.com/searches/1/edit"
      }
    }
  }
]

Show

See List. Only the hash for the requested search is returned.

Create

  • Required: search[name], search[query]
  • Optional: search[group_id] (default: All Systems or first group)

Manage Log Destinations

List

[ 
  { 
    "id":31,
    "syslog": {
      "hostname":"logs.papertrailapp.com",
      "port":11111
    },
  },

  { 
    # Another destination
  } 
]

Show

See List. Only the hash for the requested destination is returned.

Manage Users

List

[
  {
    "email":"sally@example.com",
    "id":1
  }
]

Invite

  • Required: user[email], user[read_only]

user[email]=new@example.com&user[read_only]=1

For email addresses which are not already active Papertrail accounts, an email will be sent to the address provided in email with a link to accept the invitation, create an account, and choose a password.

For existing Papertrail users, access will be granted immediately.

Retrieve account usage

List

{
  "log_data_transfer_hard_limit":1111111,
  "log_data_transfer_used":1111111,
  "log_data_transfer_used_percent":1.1111111,
  "log_data_transfer_plan_limit":111111111
}

All values are shown in bytes other than the percentage transferred. All include additional usage, if enabled, other than the plan limit. The percentage used may thus be greater than 100%. Usage represents data transferred since the start of the current billing period.

Search Events

The most basic example is simply to hit the event search API endpoint, https://papertrailapp.com/api/v1/events/search.json. Try it.

You'll receive the most recent 100 log events. You may be prompted to re-authenticate even when already logged in to the papertrailapp.com Web site.

Constrain Query

To limit results to only a specific group or system, include system_id or group_id in the query string. For example:

search.json?system_id=1234
search.json?group_id=2345

Also, system names which contain only numbers, letters, and underscores (like hostnames) can be used as arguments for system_id, in lieu of the Papertrail ID. The attribute is the Papertrail system name (name in the system JSON hash).

This can simplify linking to results for a single system, since no name-to-mapping API query is necessary. For example:

search.json?system_id=www42
search.json?system_id=my-big-server

See below for how to perform a search.

Example

Here's a live example and code from the papertrail-cli gem. Install it or read search_query.rb source.

Response

Papertrail responds with a JSON hash containing 3 important keys:

  • events: an array of hashes of log events (one hash per event)
  • min_id: the smallest event ID presented
  • max_id: the highest event ID presented

The following hash keys are defined for each log event:

  • id
  • received_at
  • generated_at
  • source_name
  • source_ip
  • facility
  • severity
  • program
  • message

A response containing no matching events will return an empty events array, such as:

{"max_id":"0","reached_beginning":true,"events":[],"min_id":"0"}

received_at and generated_at are in the time zone of the API token owner (see Profile).

To search for a specific message or string, add the optional parameter q (as in query). All search queries that work in the Papertrail Web interface should work in the API.

All parameters should be URL-encoded, as is standard for GET query strings. For example, the search string:
bob OR ("some phrase" AND sally)

would be this URL-encoded GET query string:
q=bob%20OR%20(%22some%20phrase%22%20AND%20sally)

Tail

Apps may implement a "live tail" (tail -f) style display by performing multiple successive searches for the same search query (or no search query). To do this, the max_id value from the prior result set must be passed back to the Papertrail API as the min_id parameter.

When your search query changes, omit the min_id from the prior search's result set.

Older Events

By Event ID

To obtain older events (rather than current events or tailing current events), a max_id may be passed to Papertrail. This is the newest (highest) ID that your app would like to see, and Papertrail will return the newest events that are older than that ID.

To scroll back through older events, like for a progressively-older tail, the min_id from a Papertrail response would be passed back in as the max_id. You'll receive events older than that response.

The id values for a series of events will only increase (and does indicate relative event order). id values are not sequential.

By Time

If you have a timestamp but not an event ID, pass max_time or min_time with your first request (rather than max_id or min_id). These should be in Unix time, GMT.

After the first request/response, if successive queries are needed, provide max_id as in "By Event ID" above.

Example response

Here is a response with 2 log events:

{
  "max_id":"7711582041804800",
  "min_id":"7711561783320576",
  "events":[
    {"hostname":"abc","received_at":"2011-05-18T20:30:02-07:00","severity":"Info","facility":"Cron","source_id":2,"message":"message body","program":"CROND","source_ip":"208.75.57.121","display_received_at":"May 18 20:30:02","id":7711561783320576,"source_name":"abc"},
    {"hostname":"def","received_at":"2011-05-18T20:30:02-07:00","severity":"Info","facility":"Cron","source_id":19,"message":"A short event","program":"CROND","source_ip":"208.75.57.120","display_received_at":"May 18 20:30:02","id":7711562567655424,"source_name":"server1"}
  ]
}

Papertrail uses 64-bit event IDs, which Javascript has trouble with, so the id value is a string. The other values are consistently set to the type you would expect. No values should be null except for program (when none is defined by the message). An empty message body is a blank string ('').

Submitting log messages

HTTP is very poorly suited for realtime logging. It's probably obvious from this API that we love HTTP. However, but its characteristics are almost the polar opposite of what makes an easy, resilient log sender.

Here's why HTTP doesn't do your app any favors:

  • the log submitter (your app) would need to block on a response, which slow downs its regular operations. (If it doesn't block, it needs to queue and/or accept that the messages may be lost, which is arguably worse, not to mention losing many of the benefits of HTTP)
  • in order to avoid slowing down your app, the log submission usually needs to go in a separate thread, which is non-trivial to implement. For code that's already running in an event loop, it would introduce complexity and delay to the loop.
  • scaling logging would mean queueing the log messages in your app and having one or more threads to submit them, or having to deal with backpressure.
  • there's no standard or even de facto standard

The final and biggest reason: Sending a syslog packet is extremely easy because at its core, a "syslog packet" is just a simple string (think printf). Generating and transmitting it is usually 2-4 lines of code. It's often less code and easier to follow than generating an HTTP request (and is much shorter and more elegant than HTTP log submission).

So, although we love HTTP for some things, it's a really bad fit for transmitting logs. Papertrail makes it easy to not encounter these problems.

Feel free to ask if you want a code sample for your language.

Recent Discussions

16 Apr, 2014 06:02 PM
16 Apr, 2014 03:49 PM
16 Apr, 2014 03:41 PM
16 Apr, 2014 11:12 AM
15 Apr, 2014 11:46 PM