Fastly

Fastly is a content delivery network (CDN) which caches HTTP content to improve the experience for those using your site or service. Fastly generates log messages for HTTP requests it receives and can send them in realtime to a remote syslog destination which Papertrail provides.

The result is realtime tail, search, and alerts of Fastly traffic. Along with the logs from related systems and apps all being transmitted to Papertrail, this provides a complete view of log-related activity in one place. If you aren’t familiar with Papertrail, start with a look at the event viewer.

Requirements

  1. A Fastly account. Sign up and read Fastly’s remote log streaming tutorial.

  2. A free Papertrail account.

Note: If you’re using the new Fastly interface, skip ahead to those steps.

Setup

Here’s how to aggregate HTTP request logs from Fastly in Papertrail.

  1. Create a log destination in Papertrail for Fastly logs which collects them under a single sender:

    1. Navigate to Log Destinations and click on Alternatives
    2. Click the My system’s hostname changes radio button
    3. Fill in What should we call it? on the right (“Fastly” would suffice)
    4. Click the Save button
  2. Configure Fastly to log to Papertrail:

    1. Open the Fastly app
    2. Click the Configure header link
    3. Click the Configure button
    4. Navigate to Logging
    5. Under the Papertrail endpoints section, click the New button.
    6. Provide a name (“Papertrail” would suffice)
    7. In the Address field, enter the hostname and port number from the newly created Papertrail log destination.

    For example, if the Papertrail log destination had the address logs2.papertrailapp.com:xxxxx, here’s how the resulting Fastly configuration would look.

fastly-new-papertrail-logging-endpoint.png

Advanced

Fastly uses format strings to format the log message. However, not every piece of data you may want to log is available as a format string variable. Here’s a technique for creating extensive log messages using a temporary response header.

  1. Enable VCL: Request that Fastly enable VCL for your account. This is necessary in order to remove the temporary logging header before it’s sent to clients.

  2. Create a temporary response header: Open the Fastly app, click the Configure header link, click the Configure button, navigate to Content, and click the New button under the Headers section.

    fastly-new-log-message-header

    1. Give it a Name like “Log Message.”
    2. Set Type/Action to Response/Set.
    3. Set Destination to http.X-Log-Message.
    4. Set Source to the following:

      "for=request method=" req.request " path=" req.url {" fwd=""} req.http.Fastly-Client-IP {"" status="} resp.status " bytes=" resp.http.content-length " cache=" resp.http.X-Cache " cache_hits=" resp.http.X-Cache-Hits {" timing=""} resp.http.X-Timer {"""}
      
    5. Ignore If Set and Priority can keep their default values of No and 10 respectively.
    6. Click Create to save this header.
  3. Save VCL file: Save this modified VCL to a file on your computer in order to upload it to Fastly.

  4. Upload VCL file: Navigate to VCL, and click the New button. Give this VCL a name like “Unset Log Header,” choose the saved VCL file, and click Create.

  5. Apply new response header: Navigate to Logging and edit the settings for your Papertrail destination. In the “Format String” form field, enter: resp.http.X-Log-Message. Save the change.

Deploy these changes and in a few minutes you’ll have more robust Fastly logs streamed to your Papertrail account.

Filtering logs

Often 99% or more of log messages from Fastly are successful requests for static assets like images and CSS files. In many cases, these HTTP 2xx and 3xx requests provide little operational value and are worth dropping. On the other hand, HTTP 4xx and 5xx requests are often very useful.

This filter regular expression can drop logs about requests for static assets, so that log data transfer is not consumed:

GET /.*?\.(css|png|jpg|gif|woff|ico|svg|js)|(GET|POST|HEAD) /.*? (1..|2..|3..|bingbot|Googlebot)

To apply it, visit Log Destinations, click Settings, click Add log filter, and paste the filter expression.

Fastly Next

With Fastly’s new interface, some of the steps have changed.

  1. Create a log destination in Papertrail for Fastly logs which collects them under a single sender:

    1. Navigate to Log Destinations and click on Alternatives
    2. Click the My system’s hostname changes radio button
    3. Fill in What should we call it? on the right (“Fastly” would suffice)
    4. Click the Save button
  2. Configure Fastly to log to Papertrail:

    1. Log in to the Fastly web interface and click the Configure link.
    2. From the service menu, select the appropriate service.

      service menu

    3. Click the Edit Configuration button and then select Clone active. The service version page appears.
    4. Click the Logging tab. The Logging page appears.

      logging endpoint

    5. Click the Papertrail logo. The Create a new Papertrail endpoint page appears.

      create empty

    6. In the Description field, type the name of the endpoint (Papertrail would suffice).
    7. In the Address field, type the hostname and port number from the newly created Papertrail log destination. For example, if the Papertrail log destination had the address logs2.papertrailapp.com:99999, here’s how the resulting Fastly configuration would look:

      create filled-in

Advanced

Fastly uses format strings to format the log message. However, not every piece of data you may want to log is available as a format string variable. Here’s a technique for creating extensive log messages using a temporary response header.

  1. Request Fastly enable the ability to upload custom VCL for your account by contacting support@fastly.com. You need this ability in order to remove the temporary logging header before it’s sent to clients.

  2. Create a temporary response header:

    1. Log in to the Fastly web interface and click the Configure link.
    2. From the service menu, select the appropriate service.
    3. Click the Edit configuration button and select Clone active. The service version page appears.
    4. Click the Content tab. The Content page appears.
    5. Click the Create header button. The Create a new header window appears.

    new header

    1. From the Type menu, select Response, and from the Action menu, select Set.
    2. In the Destination field, type resp.http.X-log-Message.
    3. In the Source field, type:

       "for=request method=" req.request " path=" req.url {" fwd=""} req.http.Fastly-Client-IP {"" status="} resp.status " bytes=" resp.http.content-length " cache=" resp.http.X-Cache " cache_hits=" resp.http.X-Cache-Hits {" timing=""} resp.http.X-Timer {"""}
      
    4. Leave the Ignore if set menu and the Priority field set to their default values.
    5. In the Description field, type the name of your header rule (for example, Log Message).
    6. Click the Create button to create this header. A new Date header appears on the Content page.
  3. Save this modified VCL to a file on your computer in order to upload it to Fastly.

  4. Upload the VCL you saved:

    1. Click the VCL tab. The Varnish Configurations page appears.
    2. Click the Create VCL button.

      upload vcl

    3. In the Name field, type the name of the VCL file you’re uploading (for example, Unset Log Header).
    4. Click the Upload File button and select the VCL file to upload.
    5. Click the Create button. The VCL file appears in the Varnish Configurations area.
  5. Click the Activate button to deploy your configuration changes.

In a few minutes you’ll have more robust Fastly logs streamed to your Papertrail account.

Questions

We’re happy to help. Ask a question or join our support chat.