Automatic S3 archive export

Here’s how to sign up for Amazon Web Services, create a bucket for log archives, and share write-only access to Papertrail for nightly uploads.

Sign up for Amazon Web Services

Skip this step if you already have an AWS account, like for Amazon EC2, S3, or another AWS product.

  1. Visit http://aws.amazon.com/
  2. Click Create an AWS Account (upper right)
  3. Enter your email and choose I am a new user
  4. Complete the signup form. Confirm the activation email.

Activate Amazon S3

Skip this step if your AWS account is already activated for S3.

  1. Visit http://aws.amazon.com/s3/
  2. Click Sign Up For Amazon S3
  3. Provide a credit card. You are accepting responsibility for the storage, data transfer, and requests consumed for your logs, and will be charged for it. Typically this is well under $1 (one dollar) per month.
  4. Visit http://aws.amazon.com/, click Sign In to AWS Management Console and sign in.

If a warning is displayed that your account isn’t active yet, try again in 5 minutes.

Create and share an S3 Bucket

  1. Visit http://aws.amazon.com/, click Sign In to AWS Management Console, and sign in.
  2. Click Services on the top menu, then Amazon S3, and finally Create Bucket.
  3. Fill in Bucket Name with a unique name, such as companyname-papertrail. The name should consist of only lowercase letters, numbers, and hyphens. (more info) If you have an existing bucket, you may use it too, though we recommend a bucket just for this purpose.
  4. Assign the bucket to the appropriate region.
  5. Skip the Set Properties step unless this is important for other reasons (the defaults are appropriate for Papertrail’s needs).
  6. On the Set Permissions tab: (To set up permissions with IAM, see Define Sharing Policy with IAM.)
    • Next to Access for other AWS Account, click Add Account.
    • For the user, enter 4b0a516d33d6b490d119301a4b16db3fa49a6b33bdc5135c2ab90e8184f6995f.
    • Under the Objects column, check the Write box.
    • Save the policy.
  7. Review and save.

Add User

Changes can be made after the fact by selecting the bucket and choosing the Permissions tab.

Amazon also has instructions for editing bucket permissions.

Alternative: Define sharing policy with IAM

If you followed the instructions above to grant permissions via the AWS Management Console, skip this step.

If you prefer defining a bucket policy to control access, here’s an example policy that permits Papertrail to upload archives:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PapertrailLogArchive",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::719734659904:root"
        ]
      },
      "Action": [
        "s3:DeleteObject",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::bucket-name/papertrail/logs/*"
      ]
    }
  ]
}

where bucket-name/papertrail/logs/ is the directory for Papertrail. The s3:DeleteObject action isn’t strictly necessary – it’s only used to clean up the temporary test_file.txt during initial configuration. (However, since s3:PutObject also permits overwrites, denying s3:DeleteObject doesn’t provide any significant protection.)

Tell Papertrail the bucket name

Under Settings > Archives, enable S3 archive copies and provide the S3 bucket name.

s3.png

Papertrail will perform a test upload as part of saving the bucket name (and will then delete the test file). Note that a new bucket can sometimes take several hours to become available, due to DNS propagation delays. If it fails, wait two hours, and try again.

When archives are uploaded to the bucket, each file is named under the path (key prefix) provided to Papertrail, typically papertrail/logs/<xxx> where <xxx> is an ID. For example, February 25, 2016 would be:

bucket-name/papertrail/logs/54321/dt=2016-02-25/2016-02-25.tsv.gz

Days are from midnight to midnight UTC. Alternatively, an hourly archive file for 3 PM UTC would be:

bucket-name/papertrail/logs/54321/dt=2016-02-25/2016-02-25-15.tsv.gz

Questions

Why does Papertrail support S3 but not Glacier?

Papertrail supports S3 rather than Glacier because:

  • AWS offers the ability to trickle files from S3 to Glacier using a policy that you define, so by supporting S3, Glacier is automatically a possible destination. Visit S3 Object Lifecycle Management.
  • Archived log files compress extremely well, often 15:1 or more, so the total cost of archived logs stored in S3 is extremely small (often pennies per month). Storing a long-term log archive in your S3 bucket will almost always cost less than 1% of the total cost of Papertrail. There’s effectively no cost savings.

Are Archives Encrypted at Rest?

Yes, Papertrail takes advantage of S3’s server-side encryption so that archived data is encrypted at rest using AES-256.