Stage 6 · Operate
Blameless Postmortems & Learning
Recurrence Tracking
Incident tags, repeat failure modes, action aging dashboards, and quarterly reliability themes.
Why Track Recurrence?
If the same type of incident happens twice, your postmortem process is not working. Recurrence means the action items did not prevent the problem. Tracking recurrence reveals which postmortems are effective and which are not.
When an incident recurs, it is not bad luck. It is a failure of your postmortem process to identify and implement effective preventive actions. Track recurrence to improve your process.
Incident Tags
Tag every incident with metadata that enables recurrence analysis. Tags should capture the failure mode, affected system, contributing factors, and resolution method.
incident_tags:
failure_mode:
- "database_connection_pool_exhaustion"
- "certificate_expiry"
- "disk_space_full"
- "dependency_failure"
- "configuration_error"
- "capacity_exhaustion"
- "deployment_regression"
affected_system:
- "user-api"
- "payment-service"
- "data-pipeline"
- "web-frontend"
contributing_factor:
- "missing_monitoring"
- "outdated_runbook"
- "insufficient_capacity"
- "untested_rollback"
- "manual_process"
- "single_point_of_failure"
resolution_method:
- "rollback"
- "scale_up"
- "restart"
- "configuration_change"
- "failover"
- "manual_intervention"
severity: "SEV1, SEV2, SEV3, SEV4"
detection_method:
- "automated_alert"
- "customer_report"
- "dashboard"
- "internal_report"Repeat Failure Modes
Analyze your incident history to identify repeat failure modes. If the same failure mode appears multiple times, it is a systemic issue that needs organizational attention.
failure_mode_analysis:
process:
- "Query incidents by failure_mode tag"
- "Count occurrences per failure mode"
- "Identify failure modes with count > 2"
- "Analyze contributing factors for each"
- "Identify common systemic issues"
analysis_template:
failure_mode: "database_connection_pool_exhaustion"
occurrences: 3
incidents:
- id: "INC-2024-001"
date: "2024-01-15"
service: "user-api"
contributing_factors: ["missing_monitoring", "manual_process"]
- id: "INC-2024-008"
date: "2024-03-22"
service: "payment-service"
contributing_factors: ["missing_monitoring", "insufficient_capacity"]
- id: "INC-2024-015"
date: "2024-06-10"
service: "user-api"
contributing_factors: ["missing_monitoring", "outdated_runbook"]
systemic_issues:
- "No standard connection pool monitoring across services"
- "No automated connection pool scaling"
- "No connection pool capacity review process"
organizational_action:
- "Define standard connection pool metrics"
- "Implement connection pool auto-scaling"
- "Add connection pool review to quarterly capacity planning"Action Aging Dashboard
Track action item aging to prevent items from being forgotten. An aging dashboard shows how long each action item has been open, who owns it, and whether it is blocked.
action_aging_dashboard:
panels:
- title: "Open Action Items"
type: "stat"
query: "count(action_item_status != 'verified')"
target: "Decreasing trend"
- title: "Overdue Action Items"
type: "stat"
query: "count(action_item_due < today() AND action_item_status != 'verified')"
target: "0"
alert: "If > 0, escalate to engineering manager"
- title: "Action Items by Age"
type: "bar_chart"
query: "group by age_bucket, count(action_items)"
buckets:
- "0-7 days"
- "8-14 days"
- "15-30 days"
- "31-60 days"
- "60+ days"
- title: "Action Items by Priority"
type: "table"
columns: ["Priority", "Count", "Overdue", "Avg Age (days)"]
- title: "Completion Rate"
type: "timeseries"
query: "completed_this_month / created_this_month"
target: "> 80%"
alerting:
- condition: "Overdue items > 0 for > 7 days"
action: "Escalate to engineering manager"
- condition: "Completion rate < 70% for 2 months"
action: "Review action item process"Quarterly Reliability Themes
Quarterly reliability themes focus your improvement efforts. Analyze incident patterns each quarter and choose 2-3 themes to focus on. This prevents spreading efforts too thin.
quarterly_themes:
process:
- "Review all incidents from the quarter"
- "Identify top 3 failure modes by count"
- "Select 2-3 themes for next quarter"
- "Assign theme owners"
- "Set measurable goals for each theme"
theme_examples:
- theme: "Observability Gap Closure"
goal: "Zero incidents caused by missing monitoring"
actions:
- "Audit all services for monitoring gaps"
- "Implement standard metrics for all services"
- "Verify alert coverage for all SLOs"
- theme: "Deployment Safety"
goal: "Zero deployment-caused incidents"
actions:
- "Implement automated rollback for all services"
- "Add canary deployments to all critical services"
- "Create deployment safety checklist"
- theme: "Capacity Management"
goal: "Zero capacity-related incidents"
actions:
- "Implement predictive scaling for all services"
- "Add capacity alerts to all services"
- "Conduct capacity reviews for all services"
tracking:
- "Monthly progress review"
- "Mid-quarter theme check-in"
- "End-of-quarter results review"
- "Carry forward incomplete themes"Recurrence Metrics
recurrence_metrics:
- name: "Incident Recurrence Rate"
formula: "incidents_with_prior_postmortem / total_incidents"
target: "< 10%"
frequency: "Monthly"
- name: "Failure Mode Recurrence"
formula: "repeat_failure_modes / total_failure_modes"
target: "< 20%"
frequency: "Quarterly"
- name: "Action Item Effectiveness"
formula: "incidents_prevented_by_actions / total_actions"
target: "> 80%"
frequency: "Quarterly"
- name: "Postmortem Quality Score"
criteria:
- "Blameless language used"
- "Timeline reconstructed"
- "Contributing factors identified"
- "SMART action items"
- "Reviewed within 5 business days"
target: "100%"
frequency: "Per postmortem"Recurrence tracking closes the loop on your postmortem process. It tells you whether your postmortems are actually preventing incidents. Without it, you are producing action items without knowing if they work.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.