Stage 6 · Operate
SLOs & Error Budgets
Error Budget Policy
Burn rate alerting, budget consumption, and freeze thresholds — turning reliability into decisions.
What Is an Error Budget?
An error budget is the inverse of your SLO. If your SLO is 99.9% availability, your error budget is 0.1% — the amount of unreliability you are allowed. Every minute of downtime consumes part of this budget. When the budget is exhausted, you must stop shipping changes and focus on reliability.
Error budgets are the bridge between product velocity and reliability. They give teams permission to take calculated risks. As long as you have budget, you can ship. When budget runs out, you must fix reliability first.
Burn Rate
Burn rate measures how fast you are consuming your error budget. A burn rate of 1 means you are consuming budget at exactly the expected rate. A burn rate of 2 means you will exhaust your budget twice as fast. A burn rate of 0.5 means you are consuming at half the expected rate — your reliability is better than the SLO.
# Burn rate = actual error rate / allowed error rate
# SLO: 99.9% availability (0.1% budget)
# If actual error rate is 0.3%:
burn_rate = 0.3% / 0.1% = 3.0
# You are burning budget 3x faster than allowed.
# At this rate, your 30-day budget will be exhausted in 10 days.
# If actual error rate is 0.05%:
burn_rate = 0.05% / 0.1% = 0.5
# You are burning budget at half the expected rate.
# Your budget will last longer than 30 days.Burn Rate Alerting
Burn rate alerts detect when you are consuming error budget too quickly. The key insight is that different burn rates need different response times. A very high burn rate (14.4x) will exhaust your budget in 2 days and needs an immediate page. A moderate burn rate (3x) will exhaust budget in 10 days and can wait for a ticket.
| Burn Rate | Budget Exhausted In | Alert Type | Response Time |
|---|---|---|---|
| 14.4x | 2 days | Page | Immediate |
| 6x | 5 days | Page | Within 1 hour |
| 3x | 10 days | Ticket | Next business day |
| 1x | 30 days | Dashboard | Weekly review |
# Fast burn alert - catches acute incidents (14.4x burn rate)
# Short window: 1 hour of burn at 14.4x rate
# Long window: 6 hours of burn at 14.4x rate
groups:
- name: error_budget_alerts
rules:
- record: burnrate:1h
expr: |
1 - (
sum(rate(http_request_duration_seconds_count{code!~"5.."}[1h]))
/
clamp_min(sum(rate(http_request_duration_seconds_count[1h])), 1)
) > (14.4 * 0.001)
- record: burnrate:6h
expr: |
1 - (
sum(rate(http_request_duration_seconds_count{code!~"5.."}[6h]))
/
clamp_min(sum(rate(http_request_duration_seconds_count[6h])), 1)
) > (14.4 * 0.001)
- alert: ErrorBudgetBurnRateCritical
expr: burnrate:1h and burnrate:6h
for: 2m
labels:
severity: critical
annotations:
summary: "Error budget burning at 14.4x rate"
runbook: "https://runbooks.internal/error-budget-critical"A single short window can trigger on transient spikes. The multi-window approach requires both a short window (sensitivity) and a long window (confirmation) to fire. This reduces false positives while maintaining responsiveness.
Budget Policies
Error budget policies define what actions to take at different budget levels. They should be documented, agreed upon by stakeholders, and enforced automatically when possible. A good policy removes ambiguity from decision-making during incidents.
## Error Budget Policy: user-api
### Budget Remaining: 75-100%
- Normal deployment velocity
- Feature development proceeds as planned
- No additional reliability work required
### Budget Remaining: 50-75%
- Monitor error budget burn rate closely
- Avoid high-risk deployments (schema changes, infrastructure migrations)
- Increase test coverage for error-prone paths
### Budget Remaining: 25-50%
- Require SRE sign-off for all deployments
- Prioritize reliability improvements over feature work
- Increase alerting sensitivity
- Schedule reliability sprint within 2 weeks
### Budget Remaining: 0-25%
- Deploy freeze for all non-critical changes
- All engineering effort on reliability
- Daily standup on error budget status
- SRE leadership review of all changes
### Budget Exhausted (0%)
- Mandatory rollback of recent deployments
- Incident declared on reliability
- No new features until budget recovers to 25%Freeze Thresholds
A deploy freeze is the strongest action in your error budget policy. It stops all deployment activity until reliability improves. Freezes should be triggered automatically when the budget drops below a defined threshold, typically 0-10% remaining.
groups:
- name: deploy_freeze
rules:
- record: slo:budget_remaining:ratio
expr: |
(
(0.999 - slo:http_availability:ratio_rate30d)
/
(1 - 0.999) * -1
)
- alert: DeployFreezeTriggered
expr: slo:budget_remaining:ratio < 0.10
for: 5m
labels:
severity: critical
action: deploy_freeze
annotations:
summary: "Error budget below 10% — deploy freeze activated"
description: "Remaining budget {{ $value | humanizePercentage }}"
runbook: "https://runbooks.internal/deploy-freeze"Error Budget in Prometheus
The complete error budget implementation in Prometheus records the budget ratio, burn rate, and remaining budget as separate metrics. This makes it easy to build dashboards and alerts from the same data.
groups:
- name: error_budget_recording
interval: 30s
rules:
# Error ratio over 30-day window
- record: error_budget:ratio_rate30d
expr: |
1 - (
sum(rate(http_request_duration_seconds_count{code!~"5.."}[30d]))
/
clamp_min(sum(rate(http_request_duration_seconds_count[30d])), 1)
)
# Error budget remaining (as a ratio of total budget)
- record: error_budget:remaining
expr: |
clamp((0.001 - error_budget:ratio_rate30d) / 0.001, 0, 1)
# 1-hour burn rate (for fast-burn alerts)
- record: error_budget:burn_rate:1h
expr: |
1 - (
sum(rate(http_request_duration_seconds_count{code!~"5.."}[1h]))
/
clamp_min(sum(rate(http_request_duration_seconds_count[1h])), 1)
) / 0.001
# 6-hour burn rate (for slow-burn alerts)
- record: error_budget:burn_rate:6h
expr: |
1 - (
sum(rate(http_request_duration_seconds_count{code!~"5.."}[6h]))
/
clamp_min(sum(rate(http_request_duration_seconds_count[6h])), 1)
) / 0.001Budget remaining tells you where you are now. Burn rate tells you where you are going. A service with 50% budget remaining and a 10x burn rate will exhaust its budget in 1.5 days. Always look at both metrics together.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.