Stage 4 · Provision
Observability-Driven Design
SLO-Driven Development
Defining error budgets and making reliability a first-class product feature.
What Is SLO-Driven Development?
SLO-driven development treats reliability as a feature that must be explicitly managed. Every service defines SLOs, measures SLIs, and uses error budgets to make shipping vs reliability tradeoffs. This replaces subjective debates with data-driven decisions.
Defining SLOs
SLOs should be based on user experience, not internal metrics. Users do not care about CPU usage — they care about response time and error rate. Choose SLIs that directly reflect what users experience.
api_service:
availability:
sli: "successful_requests / total_requests"
slo: 99.95%
window: 30_days
latency:
sli: "percentage of requests under 200ms"
slo: 99%
window: 30_days
data_pipeline:
freshness:
sli: "data arriving within SLA / expected data"
slo: 99.9%
window: 7_days
completeness:
sli: "records processed / records received"
slo: 99.99%
window: 7_daysAPI services focus on availability and latency. Data pipelines focus on freshness and completeness. Define SLIs that match what the service does for users.
Error Budget Policies
Error budget policies define what happens when the error budget is consumed. They should be agreed upon before the SLO is set, not during an incident. Policies create clear, predictable tradeoffs between velocity and reliability.
error_budget_policy:
- threshold: "100% remaining"
action: "Normal development, feature launches permitted"
- threshold: "50% remaining"
action: "Freeze non-critical deployments, focus on reliability"
- threshold: "25% remaining"
action: "Feature freeze, all engineering on reliability"
- threshold: "0% remaining"
action: "Complete freeze, incident review, reliability sprint"These thresholds create automatic triggers. When the error budget drops below 50%, the team shifts focus to reliability without waiting for management approval.
Burn Rate
Burn rate measures how fast the error budget is being consumed. A burn rate of 1 means you are consuming budget at the expected rate. A burn rate of 10 means you will exhaust your budget 10x faster than planned. Burn rate alerts detect SLO violations before the budget is exhausted.
# 30-day error budget: 0.05% (for 99.95% SLO)
# Budget = 0.0005 * 30_days * 24_hours * 60_minutes
# Budget = 21.6 minutes of downtime
# If you have 5 minutes of downtime in 1 hour:
burn_rate = actual_downtime / expected_downtime
burn_rate = 5 / (0.05 * 60) # 5 / 3 = 1.67
# At burn rate 1.67, budget exhaustion in:
remaining_budget = 21.6 - 5 # 16.6 minutes left
time_to_exhaustion = remaining_budget / (1.67 * 0.05 * 60)
# ≈ 33 hoursBurn rate alerts fire when the error budget is being consumed too fast, giving you time to fix the issue before the budget is exhausted.
SLO Tooling
- Google Cloud Monitoring — Built-in SLO tracking and error budgets.
- Pyrra — Open-source SLO management with Prometheus.
- Sloth — Generates Prometheus recording rules for SLOs.
- OpenSLO — Vendor-neutral SLO specification.
Organizational Alignment
SLOs align product, engineering, and business teams around reliability. When the error budget is consumed, the product team understands why features are frozen. This removes the adversarial dynamic between shipping fast and keeping things stable.
SLOs should be set by product teams, not engineering. Product decides the reliability target based on user expectations and business needs. Engineering implements the system to meet that target.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.