Stage 6 · Operate
Incident Response & RCA
Blameless RCA Process
Timeline reconstruction, contributing factors, and action items — learning from failure without blame.
Blameless Culture
A blameless RCA focuses on systems and processes, not individuals. People make mistakes because the system allowed them to. The goal is to fix the system so the same mistake cannot happen again, not to punish the person who made it.
Blameless does not mean there are no consequences. It means consequences are systemic — improve the process, add safeguards, fix the tooling. The person who made the mistake is not punished; the system that allowed the mistake is fixed.
Timeline Reconstruction
Reconstruct the incident timeline from multiple sources: chat logs, deploy events, metrics annotations, alert timestamps, and decision records. A complete timeline reveals the chain of events that led to the incident.
timeline_sources:
chat_logs:
source: "Slack #inc-* channels"
extract: "Decisions, actions, status updates"
format: "timestamp — who — what"
deploy_events:
source: "ArgoCD / GitHub Actions"
extract: "Deployments, rollbacks, config changes"
format: "timestamp — deploy id — what changed"
metrics:
source: "Grafana annotations"
extract: "Metric spikes, SLO violations"
format: "timestamp — metric — value"
alerts:
source: "PagerDuty / Alertmanager"
extract: "Alert firing, acknowledged, resolved"
format: "timestamp — alert name — severity"
decisions:
source: "Incident commander notes"
extract: "Key decisions and rationale"
format: "timestamp — decision — rationale"Five Whys
The Five Whys technique digs deeper into the root cause. Ask 'why' repeatedly until you reach a systemic cause. Stop when you reach a cause that, if fixed, would prevent the incident from recurring.
## Five Whys: API returned 503 errors
1. Why did the API return 503 errors?
→ The application ran out of database connections.
2. Why did it run out of connections?
→ The connection pool was configured for 50 connections,
but the application needed 80.
3. Why was the pool configured for 50?
→ The original configuration was set during initial deployment
and never updated as traffic grew.
4. Why was it never updated?
→ There is no monitoring for connection pool utilization,
so we did not know it was nearing capacity.
5. Why is there no monitoring?
→ We do not have a standard set of application metrics
that all services must expose.
**Root cause:** Lack of standard application metrics including
connection pool utilization.
**Action:** Define a standard metrics set and add connection
pool monitoring to all services.Do not stop at 'human error.' Human error is a symptom, not a root cause. The root cause is why the system allowed the human error to cause an incident. Keep asking why until you reach a systemic cause.
Contributing Factors
Incidents rarely have a single root cause. Contributing factors are conditions that made the incident more likely or more severe. Identifying them helps you address multiple failure points.
contributing_factors:
technical:
- "No connection pool monitoring"
- "No auto-scaling for connection pools"
- "No circuit breaker for database connections"
process:
- "No capacity review process"
- "No standard application metrics"
- "No pre-deploy validation checklist"
human:
- "Original config set by engineer who left the company"
- "No documentation of configuration rationale"
- "No handoff process for configuration ownership"
organizational:
- "No SLO for database connection availability"
- "No regular capacity planning reviews"
- "No standard onboarding for new services"Action Items
Every RCA must produce actionable items. Each item should have a clear owner, due date, and success criteria. Track action items in a shared system and follow up until they are complete.
action_items:
- id: "AI-001"
title: "Add connection pool monitoring"
owner: "platform-team"
due_date: "2024-02-15"
priority: "high"
success_criteria: |
Grafana dashboard shows connection pool utilization
for all services. Alert fires when utilization > 80%.
status: "in-progress"
- id: "AI-002"
title: "Define standard application metrics"
owner: "sre-team"
due_date: "2024-03-01"
priority: "medium"
success_criteria: |
Document defines 10 standard metrics all services
must expose. Linter enforces metric presence.
status: "not-started"
- id: "AI-003"
title: "Implement connection pool auto-scaling"
owner: "platform-team"
due_date: "2024-03-15"
priority: "high"
success_criteria: |
Connection pool scales automatically based on load.
No manual intervention needed for traffic growth.
status: "not-started"RCA Template
# Root Cause Analysis: [Incident Title]
## Metadata
- **Date:** YYYY-MM-DD
- **Severity:** SEV[X]
- **Duration:** [time]
- **Author:** [name]
- **Reviewers:** [names]
## Summary
[2-3 sentence summary of what happened, impact, and resolution]
## Impact
- **Users affected:** [number]
- **Duration:** [time]
- **Revenue impact:** [amount or "unknown"]
- **SLO impact:** [error budget consumed]
## Timeline
[Reconstructed timeline from multiple sources]
## Root Cause
[Clear description of the root cause]
## Contributing Factors
[Technical, process, human, organizational factors]
## What Went Well
[Things that helped during the incident]
## What Went Poorly
[Things that hindered the response]
## Action Items
| ID | Title | Owner | Due | Status |
|----|-------|-------|-----|--------|
| AI-001 | [title] | [owner] | [date] | [status] |
## Lessons Learned
[Key takeaways for the team]Complete the RCA within 5 business days of the incident. Delayed postmortems lose context and impact. Schedule the review meeting as soon as the draft is ready.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.