Stage 6 · Operate
Incident Response & RCA
Game Days & Fire Drills
Planned failure injection to validate incident response — practicing when the stakes are low.
What Are Game Days?
A game day is a planned exercise where you simulate a failure scenario to test your incident response capabilities. Unlike real incidents, game days have controlled conditions, safety boundaries, and a focus on learning. They reveal gaps in your processes, tooling, and team readiness.
You would not expect firefighters to perform well without training. The same applies to SRE teams. Game days are how you train for incidents. The cost of a failed game day is zero. The cost of a failed incident is real.
Game Day Types
| Type | Description | Audience | Frequency |
|---|---|---|---|
| Tabletop | Discuss scenarios on paper, no actual failures | Full team | Quarterly |
| Simulation | Simulated failure in staging environment | On-call engineers | Monthly |
| Chaos | Real failure injection in production | Experienced team | Quarterly |
| Unannounced | Surprise failure injection | Mature teams | Rare |
Planning a Game Day
A successful game day requires careful planning. Define the scenario, set safety boundaries, assign roles, and prepare rollback procedures. Never run a game day without a safety plan.
game_day_planning:
scenario:
name: "Database failover test"
description: "Simulate primary database failure"
blast_radius:
environment: "staging"
services_affected: ["user-api"]
max_duration: "30 minutes"
roles:
facilitator:
name: "[name]"
responsibility: "Run the exercise, track timeline"
attacker:
name: "[name]"
responsibility: "Inject the failure"
responder:
name: "[name]"
responsibility: "Respond as if real incident"
observer:
name: "[name]"
responsibility: "Take notes, do not participate"
safety:
abort_criteria:
- "Real production impact detected"
- "Responder requests abort"
- "Duration exceeds 30 minutes"
rollback_plan: "Restore database from snapshot"
communication: "#game-day Slack channel"
preparation:
- "Notify stakeholders 24 hours in advance"
- "Confirm staging environment health"
- "Verify monitoring and alerting working"
- "Prepare scenario injection scripts"
- "Set up observation recording"Scenario Design
Good game day scenarios are realistic, bounded, and educational. They should test specific capabilities: detection time, communication, remediation, or recovery. Design scenarios based on gaps you have identified.
scenarios:
- id: "GD-001"
name: "Database failover"
capability_tested: "Database recovery, data integrity"
difficulty: "medium"
duration: "30 minutes"
prerequisites: "Staging database with replication"
- id: "GD-002"
name: "Certificate expiry"
capability_tested: "Certificate rotation, automation"
difficulty: "easy"
duration: "15 minutes"
prerequisites: "Test domain with cert-manager"
- id: "GD-003"
name: "DNS failure"
capability_tested: "DNS redundancy, failover"
difficulty: "hard"
duration: "45 minutes"
prerequisites: "Staging environment with DNS control"
- id: "GD-004"
name: "Cascading failure"
capability_tested: "Circuit breakers, bulkheads"
difficulty: "hard"
duration: "60 minutes"
prerequisites: "Multiple dependent services in staging"
- id: "GD-005"
name: "Data corruption"
capability_tested: "Backup restoration, data recovery"
difficulty: "hard"
duration: "60 minutes"
prerequisites: "Test database with known good state"Execution
During the game day, the facilitator controls the pace. The attacker injects failures on cue. The responder works through the incident. The observer documents everything. Stick to the plan and safety boundaries.
#!/usr/bin/env bash
# Game day: Database failover simulation
set -euo pipefail
SCENARIO="database-failover"
ENVIRONMENT="staging"
LOG_FILE="gameday-$(date +%Y%m%d-%H%M%S).log"
log() {
echo "[$(date -u +%Y-%m-%dT%H:%M:%SZ)] $1" | tee -a "$LOG_FILE"
}
log "=== Game Day: $SCENARIO ==="
log "Environment: $ENVIRONMENT"
log "Start time: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
# Step 1: Verify staging is healthy
log "--- Verifying staging health ---"
kubectl get pods -n staging
curl -s http://user-api.staging:8080/healthz
# Step 2: Inject failure - stop primary database
log "--- Injecting failure: stopping primary database ---"
kubectl delete pod -l app=postgres,role=primary -n staging
# Step 3: Monitor responder
log "--- Failure injected, monitoring response ---"
log "Responder should detect and begin remediation"
# Step 4: Wait for abort or completion
log "--- Waiting for resolution or abort ---"
read -p "Press Enter to abort, or Ctrl+C to continue monitoring"Production game days require explicit approval from engineering leadership and SRE management. Start with staging. Only move to production when your team has extensive game day experience and robust safety procedures.
Debrief and Improvement
The debrief is the most important part of a game day. Review the timeline, identify gaps, and create action items. The goal is not to judge performance — it is to find systemic improvements.
# Game Day Debrief: [Scenario Name]
## Date: YYYY-MM-DD
## Participants: [names]
## Timeline
- [HH:MM] Failure injected
- [HH:MM] Responder detected issue
- [HH:MM] Responder began remediation
- [HH:MM] Resolution achieved
## Performance Metrics
- Detection time: [minutes]
- Mean time to remediate: [minutes]
- Total incident duration: [minutes]
## What Worked Well
- [Item 1]
- [Item 2]
## What Needs Improvement
- [Item 1]
- [Item 2]
## Gaps Identified
- [Gap 1: description]
- [Gap 2: description]
## Action Items
| ID | Title | Owner | Due |
|----|-------|-------|-----|
| GD-AI-001 | [title] | [owner] | [date] |
## Next Game Day
- Scenario: [next scenario]
- Date: [planned date]Share game day results with the broader engineering team. Publish a summary in a team meeting or internal blog. This builds a culture of reliability and helps other teams plan their own game days.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.