SRE
Multi-Window Multi-Burn-Rate Alerts, Explained With the Incident That Made Me Actually Understand Them
Simple threshold alerts either page you for nothing or miss real outages. Here's the incident that forced me to actually understand burn-rate math instead of copy-pasting a Prometheus rule.
For two years I ran alerts like this: "if error rate > 1% for 5 minutes, page." It felt reasonable. Then one Tuesday it paged me four times for a blip that self-resolved in ninety seconds, and three weeks later it stayed completely silent through six hours of a slow-burning outage that ate 40% of our monthly error budget before anyone noticed. Same threshold. Two opposite failures. That's when I actually sat down and understood why Google's SRE workbook insists on multi-window, multi-burn-rate alerting instead.
The incident: quiet for six hours, then a budget review nobody wanted
Our checkout service had a 99.9% availability SLO — a 30-day error budget of about 43 minutes of unavailability. A bad config push introduced an intermittent error on one code path: not enough to trip our 1%-over-5-minutes rule, but a steady 0.4% error rate that never let up. It ran for six hours before someone noticed it in a routine dashboard check, by which point it had burned roughly 35% of that month's entire error budget on a single, boring, still-unfixed bug.
A single fixed threshold can't tell the difference between a fast, severe spike and a slow, sustained leak. One needs to page you in minutes. The other needs to page you before it eats your whole month's budget — but it will never cross a 1% threshold if it never spikes.
What "burn rate" actually means
Burn rate is just: how many times faster than your budgeted rate are you consuming your error budget right now? A burn rate of 1 means you're on pace to exhaust exactly 100% of your 30-day budget in exactly 30 days — right on the edge. A burn rate of 14.4 means you're burning your entire 30-day budget in about 2 days (30 / 14.4 ≈ 2.08 days) if it keeps up. That single number turns "error rate" into "how much time until this becomes an SLO violation," which is the number that actually matters operationally.
burn_rate = (actual error rate) / (error budget rate)
error budget rate = 1 - SLO target
e.g. 99.9% SLO -> budget rate = 0.001 (0.1%)
If actual error rate = 1.4% over a window:
burn_rate = 0.014 / 0.001 = 14A burn rate of 14 means: at this rate, you'll exhaust an entire 30-day error budget in about 30/14 ≈ 2.1 days.
Why a single window always fails one direction
If you alert on a short window (say, 5 minutes) at a low burn-rate threshold, you'll page constantly on noise — a single deploy blip, a transient network hiccup, a load balancer health check flapping. If you alert on a short window at a high threshold, you'll catch severe spikes but miss exactly the slow leak that took down checkout for six hours, because 0.4% never crosses a threshold tuned for catastrophic failure. Widening the window to hours fixes the slow-leak detection but now a real severe outage takes hours to page anyone, because the long window dilutes a short severe spike into an average that looks fine.
You need at least two alerts: a fast, high-severity one (short window, high burn rate — catches sudden severe outages in minutes) and a slow, lower-severity one (long window, lower burn rate — catches exactly the slow leak that hid from us for six hours). Google's SRE workbook formalizes this as multi-window, multi-burn-rate alerting.
The rules we actually run now
For a 99.9% SLO with a 30-day budget, here's the two-tier setup we run in Prometheus. Page immediately on the fast/severe rule; route the slow/subtle rule to a ticket or a Slack channel someone actually checks daily — not a 2am page.
| Severity | Burn rate | Long window | Short window (for reset detection) | Budget consumed if sustained |
|---|---|---|---|---|
| Page (fast) | 14.4× | 1 hour | 5 minutes | 2% of 30-day budget in 1 hour |
| Ticket (slow) | 1× | 3 days | 6 hours | 10% of 30-day budget in 3 days |
- alert: CheckoutSLOFastBurn
expr: |
(
sum(rate(checkout_requests_total{code=~"5.."}[1h]))
/
sum(rate(checkout_requests_total[1h]))
) > (14.4 * 0.001)
and
(
sum(rate(checkout_requests_total{code=~"5.."}[5m]))
/
sum(rate(checkout_requests_total[5m]))
) > (14.4 * 0.001)
for: 2m
labels:
severity: page
annotations:
summary: "Checkout burning error budget 14.4x faster than sustainable"Both windows must agree the burn rate is high — the short window confirms the spike is still happening right now (not a historical blip that already recovered), the long window confirms it's not just a single-scrape fluke.
What actually changed after we shipped this
- The slow-leak class of incident (like our six-hour checkout bug) now surfaces within about 3 hours instead of being found by accident during a dashboard review.
- Page volume for genuinely transient blips dropped by roughly 70%, because the fast tier requires both a short and long window to agree — a single 90-second blip no longer sustains long enough to trip both.
- Postmortems now cite an actual burn-rate number instead of "error rate was elevated for a while," which makes the retro concrete: we know exactly how much budget was spent and exactly how fast.
None of this is exotic — it's the standard approach in the Google SRE workbook. But I didn't really *understand* why until I'd lived through the failure mode it's specifically designed to prevent. If your alerting is still a single threshold on a single window, the question worth asking isn't "is my threshold right" — it's "which of these two failure modes am I currently blind to."