Fastly

Fastly is a content delivery network (CDN) that caches HTTP content to improve the experience for those using your site or service. Fastly generates log messages for HTTP requests it receives and can send them in realtime to a Papertrail remote syslog destination.

The result is realtime tail, search, and alerts on Fastly traffic. When logs from related systems and apps are also being transmitted to Papertrail, this provides a complete view of logged activity in one place. If you aren’t familiar with Papertrail, start with a look at the event viewer.

Requirements

  1. A Fastly account. Sign up and read Fastly’s remote log streaming tutorial.

  2. A free Papertrail account.

Setup

Here’s how to aggregate HTTP request logs from Fastly in Papertrail.

  1. Create a log destination in Papertrail for Fastly logs that collects them under a single sender:

    1. Navigate to Log Destinations and click on Alternatives.
    2. Click My system’s hostname changes.
    3. Fill in What should we call it? on the right (for example, Fastly).
    4. Click Save.
  2. Configure Fastly to log to Papertrail:

    1. Log in to the Fastly web interface and click Configure.
    2. From the service menu, select the appropriate service.

      service menu

    3. Click Edit Configuration and select Clone active. The service version page appears.
    4. Click the Logging tab. The Logging page appears.

      logging endpoint

    5. Click the Papertrail logo. The Create a new Papertrail endpoint page appears.

      create empty

    6. In the Description field, enter the name of the endpoint (for example, Papertrail).
    7. In the Address field, enter the hostname and port number from the Papertrail log destination created in Step 1. For example, if the Papertrail log destination had the address logs2.papertrailapp.com:99999, here’s how the resulting Fastly configuration would look:

      create filled-in

Filtering logs

Often 99% or more of log messages from Fastly are successful requests for static assets like images and CSS files. In many cases, these HTTP 2xx and 3xx requests provide little operational value and are worth dropping. On the other hand, requests with HTTP 4xx and 5xx statuses are often very useful.

This filter regular expression can drop logs about requests for static assets, so that log data transfer is not consumed:

GET /.*?\.(css|png|jpg|gif|woff|ico|svg|js)|(GET|POST|HEAD) /.*? (1..|2..|3..|bingbot|Googlebot)

To apply it, visit Log Destinations, click Settings on the Fastly destination, click Add log filter, and paste the filter expression.

Advanced

Fastly uses format strings to format the log message. However, not every piece of data you may want to log is available as a format string variable. Here’s a technique for creating extensive log messages using a temporary response header.

  1. Request Fastly enable the ability to upload custom VCL for your account by contacting support@fastly.com. You need this ability in order to remove the temporary logging header before it’s sent to clients.

  2. Create a temporary response header:

    1. Log in to the Fastly web interface and click the Configure link.
    2. From the service menu, select the appropriate service.
    3. Click Edit configuration and select Clone active. The service version page appears.
    4. Click the Content tab. The Content page appears.
    5. Click Create header. The Create a new header window appears.

    new header

    1. From the Type menu, select Response, and from the Action menu, select Set.
    2. In the Destination field, type resp.http.X-log-Message.
    3. In the Source field, enter:

       "for=request method=" req.request " path=" req.url {" fwd=""} req.http.Fastly-Client-IP {"" status="} resp.status " bytes=" resp.http.content-length " cache=" resp.http.X-Cache " cache_hits=" resp.http.X-Cache-Hits {" timing=""} resp.http.X-Timer {"""}
      
    4. Leave the Ignore if set menu and the Priority field set to their default values.
    5. In the Description field, type the name of your header rule (for example, Log Message).
    6. Click Create to create this header. A new Date header appears on the Content page.
  3. Save this modified VCL to a file on your computer in order to upload it to Fastly.

  4. Upload the VCL you saved:

    1. Click the VCL tab. The Varnish Configurations page appears.
    2. Click Create VCL.

      upload vcl

    3. In the Name field, type the name of the VCL file you’re uploading (for example, Unset Log Header).
    4. Click Upload File and select the VCL file to upload.
    5. Click Create. The VCL file appears in the Varnish Configurations area.
  5. Click Activate to deploy your configuration changes.

In a few minutes you’ll have more robust Fastly logs streamed to your Papertrail account.

Questions

We’re happy to help. Ask a question or join our support chat.