Stage 4 · Provision
Observability-Driven Design
Alerting Philosophy
Symptom-based vs cause-based alerts, reducing alert fatigue, and building actionable alerting.
Purpose of Alerting
Alerting exists to notify humans about problems that require human intervention. If an alert fires but no action is needed, it is noise. Every alert should require a meaningful response. If no response is needed, it should not be an alert.
Symptom-Based Alerts
Symptom-based alerts fire when users are affected. They measure what users experience: error rate, latency, throughput. These alerts are always actionable because they indicate a user-facing problem.
alerts:
- name: "HighErrorRate"
expr: "rate(http_requests_total{status=~'5..'}[5m]) / rate(http_requests_total[5m]) > 0.01"
message: "Error rate above 1% for 5 minutes"
- name: "HighLatency"
expr: "histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 0.5"
message: "p99 latency above 500ms for 5 minutes"
- name: "LowThroughput"
expr: "rate(http_requests_total[5m]) < 100"
message: "Request rate dropped below expected baseline"These alerts fire when users are actually experiencing problems. They are always actionable and never fire for internal issues that do not affect users.
Cause-Based Alerts
Cause-based alerts fire when an internal component is unhealthy: CPU > 90%, disk > 80%, pod restarting. These are useful for capacity planning and proactive maintenance but should not page humans unless they affect users.
CPU at 90% is not an emergency unless it causes user-facing errors. Disk at 80% is a warning, not a page. Cause-based alerts should be ticketed, not paged.
Reducing Alert Fatigue
- Delete alerts that never require action — If no one acts, delete it.
- Combine related alerts — One alert for a service, not ten for each metric.
- Use severity levels — Page for user-facing issues, ticket for capacity.
- Tune thresholds — Alerts that fire daily become invisible.
- Auto-resolve — If the condition clears, auto-resolve the alert.
Alert Hierarchy
| Level | Response | Example |
|---|---|---|
| Page | Wake someone up | Service error rate > 5% |
| Ticket | Fix during business hours | Disk usage > 80% |
| Log | Review in retrospective | Certificate expires in 30 days |
| Dashboard | Monitor visually | CPU usage trending up |
Pager Rotation Best Practices
- Rotate weekly — Shorter rotations prevent burnout.
- Handoff meetings — 15-minute handoff between outgoing and incoming on-call.
- Follow-the-sun — Distribute on-call across time zones.
- Secondary on-call — Always have a backup for the primary.
- Compensate on-call — On-call duty is work, compensate accordingly.
When engineers receive 50+ alerts per day, they stop paying attention. The one alert that matters gets buried. Aggressive alert hygiene is a reliability investment, not overhead.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.