Stage 6 · Operate
Blameless Postmortems & Learning
Timeline Reconstruction
Chat logs, deploy events, metrics annotations, alert timestamps, and decision records.
Why Reconstruct the Timeline?
A timeline is the factual backbone of your postmortem. It shows what happened, when it happened, and what was done. Without a timeline, postmortem discussions become opinions and blame. With a timeline, they become evidence-based learning.
A timeline records events as they happened, not as people remember them. Memory is unreliable, especially under stress. Use data sources to build the timeline, not human recollection.
Timeline Sources
timeline_sources:
chat_logs:
source: "Slack #inc-* channels"
extract: "Decisions, actions, status updates"
format: "timestamp — who — what"
tool: "Slack export or search"
deploy_events:
source: "ArgoCD / GitHub Actions / CI/CD"
extract: "Deploys, rollbacks, config changes"
format: "timestamp — deploy id — what changed"
tool: "ArgoCD events, GitHub audit log"
metrics:
source: "Grafana annotations"
extract: "Metric spikes, SLO violations"
format: "timestamp — metric — value"
tool: "Grafana annotation API"
alerts:
source: "PagerDuty / Alertmanager"
extract: "Alert firing, acknowledged, resolved"
format: "timestamp — alert name — severity"
tool: "PagerDuty API, Alertmanager API"
kubernetes_events:
source: "kubectl get events"
extract: "Pod scheduling, scaling, crashes"
format: "timestamp — event — reason"
tool: "kubectl, Kubernetes audit logs"
application_logs:
source: "ELK / Loki / CloudWatch"
extract: "Errors, warnings, key events"
format: "timestamp — level — message"
tool: "Log aggregation system"
decisions:
source: "Incident commander notes"
extract: "Key decisions and rationale"
format: "timestamp — decision — rationale"
tool: "War room channel, scribe notes"Assembling the Timeline
Combine all sources into a single chronological timeline. Use UTC for all timestamps. Include both technical events and human decisions. The timeline should tell the complete story of the incident.
assembly_process:
step_1_collect:
- "Export chat logs from war room channel"
- "Export deploy events from CI/CD"
- "Export alert events from PagerDuty"
- "Export Kubernetes events"
- "Export relevant metrics annotations"
step_2_normalize:
- "Convert all timestamps to UTC"
- "Sort all events chronologically"
- "Remove duplicates"
- "Label each event with its source"
step_3_contextualize:
- "Add context to technical events"
- "Mark decision points"
- "Note who was involved"
- "Link to relevant evidence"
step_4_review:
- "Have participants verify accuracy"
- "Fill in gaps with additional data"
- "Correct any errors"
- "Finalize the timeline"Decision Points
Mark decision points in the timeline. These are moments where someone chose an action. Understanding why decisions were made helps identify systemic improvements, not individual mistakes.
decision_points:
- timestamp: "2024-01-15T14:30:00Z"
decision: "Declare SEV1 incident"
made_by: "IC: Alice"
rationale: "Error rate > 50% for 2 minutes"
alternatives:
- "Wait and monitor (rejected: impact too high)"
evidence: "Grafana dashboard showing error spike"
- timestamp: "2024-01-15T14:35:00Z"
decision: "Rollback to v1.2.2"
made_by: "IC: Alice, Ops: Bob"
rationale: "Recent deploy correlates with error spike"
alternatives:
- "Scale up (rejected: does not address root cause)"
- "Restart pods (rejected: may not help if deploy is cause)"
evidence: "Deploy history showing v1.2.3 at 14:20"
- timestamp: "2024-01-15T14:45:00Z"
decision: "Notify VP Engineering"
made_by: "IC: Alice"
rationale: "SEV1 exceeds executive notification threshold"
alternatives: []
evidence: "Escalation policy requiring SEV1 notification"Timeline Template
## Incident Timeline: [Incident Title]
### Pre-Incident Context
- **Time period:** [start of relevant context]
- **System state:** [normal/degraded/other incidents]
### Timeline
| Time (UTC) | Event | Source | Notes |
|------------|-------|--------|-------|
| 14:20 | Deploy v1.2.3 rolled out | ArgoCD | Normal deployment |
| 14:25 | Error rate starts increasing | Prometheus | Slowly at first |
| 14:30 | Error rate exceeds 5% | Alertmanager | Alert fires |
| 14:30 | IC declares SEV1 incident | Slack | Alice takes IC role |
| 14:32 | War room opened | Slack | #inc-2024-0115 |
| 14:35 | Decision: rollback to v1.2.2 | Slack | Alice decides with Bob |
| 14:37 | Rollback initiated | ArgoCD | Rollback in progress |
| 14:42 | Rollback complete | ArgoCD | New version deployed |
| 14:45 | Error rate decreasing | Prometheus | Recovery starting |
| 14:50 | Error rate below 1% | Prometheus | Incident resolved |
| 14:55 | Incident declared resolved | Slack | Alice declares resolution |
### Post-Incident
| Time (UTC) | Event | Source | Notes |
|------------|-------|--------|-------|
| 15:00 | Status page updated | Statuspage | Resolution posted |
| 15:30 | Postmortem scheduled | Calendar | Tomorrow 10 AM |
| 16:00 | Initial findings shared | Email | Stakeholders notified |Common Pitfalls
- Relying on memory — Use data sources, not recollection.
- Missing decision points — Mark every significant decision.
- Inconsistent timestamps — Use UTC for all timestamps.
- Missing context — Explain why events matter, not just what happened.
- Incomplete sources — Gather data from all available sources.
- Blaming individuals — Record what happened, not who failed.
Build the timeline within 24 hours of the incident. Memory degrades quickly. Data sources may be rotated or deleted. The sooner you build the timeline, the more complete and accurate it will be.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.