Stage 6 · Operate
Incident Response & On-Call
Incident Commander Playbook
IC, operations, communications, and scribe roles with clear handoff checkpoints.
The IC Role
The Incident Commander owns the incident, not the fix. The IC coordinates the response, makes decisions, and ensures communication flows. The IC does not debug — debugging is the operations role. The IC's job is to make the incident end as fast as possible.
The most common IC mistake is jumping into debugging. The IC must resist this urge. The IC's value is in coordination, decision-making, and communication. Let the operations engineer debug.
IC Responsibilities
ic_responsibilities:
declaration:
- "Assess impact and assign severity"
- "Declare incident in Slack and PagerDuty"
- "Open war room channel"
role_assignment:
- "Assign operations lead"
- "Assign communications lead"
- "Assign scribe"
- "Verify all roles are filled"
coordination:
- "Track progress toward resolution"
- "Ensure communication cadence is maintained"
- "Remove blockers for the team"
- "Make escalation decisions"
decision_making:
- "Approve rollback decisions"
- "Approve escalation to SEV1"
- "Approve communications to stakeholders"
- "Decide when to declare resolved"
closure:
- "Verify impact is resolved"
- "Confirm monitoring shows normal behavior"
- "Declare incident resolved"
- "Assign postmortem owner"Supporting Roles
| Role | Responsibility | Reports To |
|---|---|---|
| Incident Commander | Owns incident, coordinates, decides | N/A — leads the incident |
| Operations Lead | Technical investigation and remediation | Incident Commander |
| Communications Lead | Internal and external status updates | Incident Commander |
| Scribe | Documents timeline and decisions in real-time | Incident Commander |
Handoff Checkpoints
The IC role may need to be handed off during long incidents. Define clear checkpoints for when and how to hand off. This prevents the IC from becoming a bottleneck during extended incidents.
handoff_rules:
when_to_handoff:
- "IC has been on for > 2 hours continuously"
- "IC needs sleep (shift change)"
- "IC is no longer the best person to lead"
- "IC requests handoff due to stress"
handoff_process:
- "Current IC briefs incoming IC on status"
- "Transfer war room ownership"
- "Transfer communication responsibility"
- "Update incident channel with IC change"
- "Incoming IC confirms understanding"
handoff_checklist:
- "Active incidents documented"
- "Current hypotheses and actions listed"
- "Communication cadence established"
- "Escalation status documented"
- "All team members introduced to new IC"IC Decision-Making
The IC makes decisions under uncertainty. Use a structured decision framework: gather facts, consider options, make a call, communicate the decision, and move on. Do not let perfect be the enemy of good.
decision_framework:
step_1_gather_facts:
- "What do we know for certain?"
- "What are we unsure about?"
- "What is the current impact?"
step_2_consider_options:
- "What are our options?"
- "What are the risks of each option?"
- "What is the fastest path to mitigation?"
step_3_make_call:
- "Choose the option with best risk/reward"
- "Do not wait for perfect information"
- "Document the decision and rationale"
step_4_communicate:
- "Announce the decision to the team"
- "Update stakeholders if needed"
- "Ensure everyone understands their role"
step_5_move_on:
- "Do not second-guess the decision"
- "Focus on next action"
- "Revisit if new information changes the picture"Most incident decisions are reversible. A rollback can be re-deployed. A scale-up can be scaled down. A status update can be corrected. Make the decision quickly and move on. Speed matters more than perfection.
IC Checklist
## IC Checklist: Incident Declaration
### Immediate Actions (First 5 minutes)
- [ ] Acknowledge alert within 5 minutes
- [ ] Assess impact: who is affected, how many, how badly
- [ ] Assign severity level
- [ ] Open war room channel: #inc-YYYY-MM-DD-description
- [ ] Declare incident in channel
### Role Assignment (First 10 minutes)
- [ ] Assign Operations Lead
- [ ] Assign Communications Lead
- [ ] Assign Scribe
- [ ] Verify all roles are acknowledged
### Coordination (Ongoing)
- [ ] Set communication cadence (every 15-30 min)
- [ ] Track progress toward mitigation
- [ ] Remove blockers for the team
- [ ] Make escalation decisions
- [ ] Update stakeholders
### Resolution (When ready)
- [ ] Verify impact is resolved
- [ ] Confirm monitoring shows normal behavior
- [ ] Get operations lead sign-off
- [ ] Declare incident resolved
- [ ] Assign postmortem owner
- [ ] Close war room channel
### Post-Incident (Within 24 hours)
- [ ] Schedule postmortem meeting
- [ ] Assign postmortem author
- [ ] Share initial findings with stakeholdersIC skills improve with practice. Use game days to practice incident command in a low-stakes environment. Rotate the IC role so multiple team members can lead incidents.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.