Heroku

Intro

Heroku is a Platform-as-a-Service for deploying and scaling applications. Here’s how to aggregate logs from a Heroku app. New to Papertrail or Heroku? This introduction explains how great log visibility helps.

Setup

Heroku apps can log to Papertrail using either of 2 methods. Both capture all Heroku logs, including router output, app/dyno output, platform errors, Heroku Postgres status, and any other log messages.

Method 1: Add-on

Papertrail has a Heroku add-on for one-click hosted log management. Activate it on elements.heroku.com.

Multiple Heroku apps

To have more than 1 Heroku app log to Papertrail, determine whether the additional app(s) are managed by, accessed by, and paid for by the same team as the existing app.

For example, a small ops team which maintains myapp-production and myapp-staging apps might prefer to have 1 Papertrail add-on subscription capture logs from both apps. On the other hand, a larger company (with app-specific access policies or payment methods) or a consultancy (managing apps for different clients) would want each Papertrail subscription to serve only 1 app.

Method 2: Standalone

New (2015-12-31): Send logs from multiple Heroku apps to 1 standard Papertrail log destination. If you have activated Heroku drains before December 31, we’ve made the setup (below) easier.

1. Create a Papertrail account (free)

If you don’t already have an account, sign up (free).

2. Setup Heroku drain

Visit Add Systems. You’ll see a log destination hostname and port. Add the Papertrail destination as a Heroku log drain:

heroku drains:add syslog+tls://logsN.papertrailapp.com:XXXXX

Replace logsN and XXXXX with the hostname and port from the page above. For example:

heroku drains:add syslog+tls://logs2.papertrailapp.com:12345 --app my-app-42

Wait 30 seconds for logging to start. Once it starts, messages will appear in Papertrail’s event viewer in realtime. Use heroku drains:add to add the same drain to any additional apps.

3. Name the app(s)

Because Papertrail has no way to obtain the app’s name, its logs will be attributed to a Heroku-generated GUID like d.1234-5678. Give the app a more meaningful name.

This command will list all drains attached to all Heroku apps, including the drain ID:

heroku apps | grep -v "=" | cut -d " " -f 1 | xargs -tL 1 heroku drains --app

On Papertrail’s Dashboard, click the “All Systems” group name link. Scroll down to the new drain ID and click “Settings.” Change the sender’s name from the drain ID to a more useful name (such as the Heroku app name). Repeat for each new app/drain.

Note: A Papertrail sender will only be created after the first log message is received from Heroku.

4. Add Heroku searches (optional)

Papertrail provides Heroku-specific searches when using the add-on, but these aren’t created on standalone accounts. Copy and paste the following commands to add them. Replace YOUR_TOKEN with your API token.

curl -G -v -H "X-Papertrail-Token: YOUR_TOKEN" -X POST https://papertrailapp.com/api/v1/searches.json --data-urlencode 'search[name]=Platform errors' --data-urlencode 'search[query]="error code=H" OR "Error R" OR "Error L"'

curl -G -v -H "X-Papertrail-Token: YOUR_TOKEN" -X POST https://papertrailapp.com/api/v1/searches.json --data-urlencode 'search[name]=Deploys' --data-urlencode 'search[query]=program:(heroku/api heroku/slug) -scheduler'

curl -G -v -H "X-Papertrail-Token: YOUR_TOKEN" -X POST https://papertrailapp.com/api/v1/searches.json --data-urlencode 'search[name]=Scheduler jobs' --data-urlencode 'search[query]=program:scheduler'

curl -G -v -H "X-Papertrail-Token: YOUR_TOKEN" -X POST https://papertrailapp.com/api/v1/searches.json --data-urlencode 'search[name]=Dyno state changes' --data-urlencode 'search[query]=web (Idling OR Unidling OR Cycling OR "State changed" OR "Starting process")'

curl -G -v -H "X-Papertrail-Token: YOUR_TOKEN" -X POST https://papertrailapp.com/api/v1/searches.json --data-urlencode 'search[name]=Web app output' --data-urlencode 'search[query]="app/web"'

Notes

Unicorn

If you have added unicorn to your Heroku app stack, Papertrail’s Unicorn logging explanation may be useful.

L10 errors

Logplex may log the following message:

heroku logplex Error L10 (output buffer overflow): 500 messages dropped since 2013-04-17T19:04:46+00:00.

L10 means that Logplex could not keep up with the volume of logs generated by the application and has been forced to discard some messages to a specific drain. Heroku’s error codes has more.

L10 doesn’t mean that Papertrail is responding slowly. It means that the sending service has overwhelmed logplex with an extremely high volume of messages, on the order of thousands of messages per second (from a single dyno or worker, not app-wide). That volume momentarily overwhelms logplex itself.

Note that the very high log volume may only exist for a moment, like because a dyno tried to log the contents of a POSTed file.

The root cause is almost always unintentionally logging a ton of data, like binary data from a form upload or unanticipated output from a background job. If you don’t see cases of this, contact us and we’ll help track it down.

Message Formatting

Papertrail displays the Heroku app name as the sender name. With the Heroku add-on in method 1, this is the actual Heroku app name (such as electric-fog-9847). With a standalone account in method 2, it is the system name value that you provide, which may be the app name or another string of your choosing.

In Papertrail’s webhooks API, this is passed as the source_name key.

Papertrail displays the Heroku process name in the Papertrail program name. A few examples are: app/web.2, heroku/slugc, or heroku/nginx. See heroku ps output for additional process names.

In Papertrail’s webhooks API, this is passed as the program key.

Troubleshooting

Rake messages don’t appear in the event viewer

Short lived dynos only log their output to the console so only startup and shutdown messages are displayed in Papertrail. For Rake tasks, the workaround is to run in detached mode using:

heroku run:detached rake

rather than:

heroku run rake

Logs generated by a Node.js application aren’t colorized

Add the --color switch to the app’s Procfile and set it to always. For example:

web: node index.js --color=always

Metering & Pricing

Heroku’s add-on system pro-rates add-on subscription charges, so an add-on subscription or plan choice may only be active for a few hours or a few days. For that reason, Papertrail’s Heroku add-on measures logs on a daily basis; measuring usage over a month-long period would not work.

In contrast, Papertrail’s standalone service measures logs on a monthly basis. Log data transfer above the plan limit may be purchased by enabling additional usage, which will provide an extra 200% of data on a pay-as-you-go model. For example, with additional usage enabled, an 8 GB plan can provide up to 8 GB + 16 GB = 24 GB.

The cost for a given log volume is intended to be equivalent between Papertrail’s Heroku add-on and a standalone account. Neither is intentionally more or less expensive. The total cost may be different based on your app’s usage patterns.

Heroku add-on usage resets daily at midnight (UTC). Paid standalone accounts reset each month on the day that service was activated.

Migrating

To migrate from one type of service to another, such as from a add-ons to a standalone account or vice versa, follow the instructions for the type of service you’ll be migrating to.

After setting up the new service, confirm it’s working. Once you see logs on the new Papertrail service, de-provision the old Papertrail service. This typically means deactivating Heroku add-on(s) with heroku addons:remove or removing standalone drain(s) with heroku drains:remove.