Log flood detection

Log rate spikes are common and often go unnoticed. They could be an indication that something went terribly wrong or a high-traffic system was unintentionally configured with verbose logging. Papertrail can notify you if your log rate is higher than expected.

How is this useful?

Here are a few real situations Papertrail customers have experienced which we realized could be solved by this single, simple notification.

• Drastic changes in usage patterns: A database went offline and all connected clients generated error messages.
• Middle-of-the-night jobs: An unattended background job unintentionally logged dozens of messages. Thousands of these jobs ran every night, and it went unnoticed for months because no one happened to look at the logs while it was running.
• Forgotten verbose logging: One customer was troubleshooting a firewall problem and enabled iptables logging of every packet. Great idea, but it didn’t get disabled until they found it a few hours later.

In any of these situations, it would have been helpful to know at the time the change occurred instead of being surprised much later.

When log velocity significantly increases, if the messages are not useful to you, feel free to filter certain messages or temporarily mute certain senders while troubleshooting.

Putting it into practice

Set a log rate on your account and Papertrail will notify everyone interested in usage notifications if it’s been exceeded for at least 10 minutes. System reboots or other momentary spikes won’t trigger this notification. We’ll send an email at most every 6 hours linking directly to the logs at that point in time so you can see what was happening.

Papertrail is in a unique position to call attention to things that are clearly unusual. It’s our goal to give you better visibility into your logs and usage.