Stage 6 · Operate
SLOs & Production Observability
SLIs, SLOs, and Error Budgets
Defining availability and latency SLIs, objective windows, budget policies, and user-journey targets.
Service Level Indicators
SLIs are quantitative measures of service behavior from the user's perspective. They represent the key signals that matter to users: availability, latency, throughput, and correctness. An SLI is a ratio of good events to total events.
# Availability SLI (success ratio)
1 - (sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])))
# Latency SLI (fraction of requests under threshold)
sum(rate(http_request_duration_seconds_bucket{le="0.2"}[5m]))
/ sum(rate(http_request_duration_seconds_count[5m]))Service Level Objectives
SLOs define the target value for an SLI over a specified time window. An SLO of 99.9% availability over a 30-day window means no more than 0.1% of requests can fail in that period. SLOs set expectations for reliability.
| SLO | Window | Error Budget (30 days) |
|---|---|---|
| 99.9% | 30 days | 43.2 minutes of downtime |
| 99.95% | 30 days | 21.6 minutes of downtime |
| 99.99% | 30 days | 4.3 minutes of downtime |
SLOs represent the minimum reliability users need. They are not aspirational targets. An SLO of 99.9% means you accept 0.1% failure. Setting SLOs too high wastes engineering effort on diminishing returns.
Error Budgets
Error budgets are the inverse of SLOs. If your SLO is 99.9%, your error budget is 0.1%. The budget represents how much failure is acceptable. When the budget is depleted, you stop shipping features and focus on reliability.
# Error budget consumed over 30 days
1 - (
sum(increase(http_requests_total{status!~"5.."}[30d]))
/ sum(increase(http_requests_total[30d]))
) / 0.001
# Error budget remaining percentage
(0.001 - (
1 - sum(increase(http_requests_total{status=~"5.."}[30d]))
/ sum(increase(http_requests_total[30d]))
)) / 0.001 * 100Budget Policies
Budget policies define what happens when the error budget is consumed. They create a feedback loop between reliability and velocity. Common policies range from freezing deployments to requiring additional reliability reviews.
| Budget Remaining | Policy |
|---|---|
| 100% - 50% | Normal development, no restrictions |
| 50% - 25% | Increased review, reliability PRs prioritized |
| 25% - 10% | Feature freeze, focus on reliability |
| 10% - 0% | Only critical fixes, incident response mode |
User Journey SLIs
Define SLIs for critical user journeys, not individual services. A checkout flow may involve 5 microservices. The SLI measures the end-to-end success rate of the checkout, not each service independently.
# Checkout success rate (end-to-end)
sum(rate(checkout_completed_total[5m]))
/ sum(rate(checkout_initiated_total[5m]))Implementation with Prometheus
groups:
- name: slo_rules
rules:
- record: slo:error_ratio:rate5m
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m]))
- record: slo:error_budget:remaining30d
expr: |
(0.001 - (
1 - sum(increase(http_requests_total{status=~"5.."}[30d]))
/ sum(increase(http_requests_total[30d]))
)) / 0.001 * 100Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.