Stage 6 · Operate
Incident Response & On-Call
Severity Levels
SEV definitions, customer impact thresholds, escalation triggers, and executive notifications.
Severity Framework
Severity levels determine how fast you respond, who gets involved, and how you communicate. A consistent severity framework eliminates confusion during incidents and ensures the response matches the impact.
A database corruption affecting 1 user is not a SEV1. A full service outage affecting all users is. Base severity on customer impact, not on how interesting the bug is.
SEV Definitions
severity_levels:
SEV1:
name: "Critical"
description: "Complete service outage or data loss affecting all users"
impact: "All users unable to use the service"
response_time: "Immediate (within 5 minutes)"
communication: "Every 15 minutes"
resolution_target: "30 minutes"
examples:
- "Complete API outage"
- "Data breach confirmed"
- "Payment processing failure for all users"
- "Data loss or corruption"
SEV2:
name: "Major"
description: "Significant service degradation affecting many users"
impact: "Majority of users experiencing errors or severe degradation"
response_time: "Within 15 minutes"
communication: "Every 30 minutes"
resolution_target: "2 hours"
examples:
- "Error rate > 10% for specific endpoint"
- "Latency p99 > 5s for all users"
- "Authentication service degraded"
- "Single region outage (multi-region service)"
SEV3:
name: "Minor"
description: "Service degradation affecting some users"
impact: "Subset of users experiencing issues"
response_time: "Within 1 hour"
communication: "Every hour"
resolution_target: "8 hours"
examples:
- "Error rate > 5% for specific endpoint"
- "Latency p99 > 2s for some users"
- "Non-critical feature broken"
- "Single availability zone issue"
SEV4:
name: "Low"
description: "Minor issue with minimal user impact"
impact: "Few users affected, workaround available"
response_time: "Within 4 hours"
communication: "Daily"
resolution_target: "24 hours"
examples:
- "Intermittent errors for < 1% of users"
- "Minor UI bug"
- "Documentation error"Impact Mapping
Map specific impacts to severity levels. This removes ambiguity from severity assignment. When an alert fires, the impact determines the severity automatically.
impact_severity_map:
error_rate:
threshold: "5% for 5 minutes"
severity: "SEV3"
escalation: "PagerDuty primary"
threshold: "10% for 5 minutes"
severity: "SEV2"
escalation: "PagerDuty primary + secondary"
threshold: "50% for 2 minutes"
severity: "SEV1"
escalation: "PagerDuty all hands"
latency:
threshold: "p99 > 2s for 5 minutes"
severity: "SEV3"
escalation: "PagerDuty primary"
threshold: "p99 > 5s for 5 minutes"
severity: "SEV2"
escalation: "PagerDuty primary + secondary"
availability:
threshold: "< 99.9% for 15 minutes"
severity: "SEV3"
escalation: "PagerDuty primary"
threshold: "< 99% for 5 minutes"
severity: "SEV2"
escalation: "PagerDuty primary + secondary"
threshold: "< 50% for 2 minutes"
severity: "SEV1"
escalation: "PagerDuty all hands"Escalation Triggers
Define triggers that automatically escalate severity. If an incident is not resolved within the resolution target, escalate. If the impact is expanding, escalate. These triggers prevent stuck incidents.
escalation_triggers:
time_based:
- condition: "SEV3 not resolved in 4 hours"
action: "Escalate to SEV2"
notify: "SRE manager"
- condition: "SEV2 not resolved in 2 hours"
action: "Escalate to SEV1"
notify: "VP engineering"
impact_based:
- condition: "Error rate increasing despite mitigation"
action: "Increase severity by 1"
notify: "Incident commander"
- condition: "Incident spreading to additional services"
action: "Escalate to SEV1"
notify: "All engineering leads"
manual:
- condition: "IC declares escalation"
action: "Escalate immediately"
notify: "Per escalation policy"Executive Notifications
Executives need to know about major incidents but do not need every detail. Define clear triggers for executive notification and provide concise, impact-focused updates.
executive_notifications:
triggers:
- severity: "SEV1"
timing: "Immediately upon declaration"
recipients: ["VP Engineering", "CTO"]
channel: "PagerDuty + Slack DM"
- severity: "SEV2"
timing: "Within 30 minutes"
recipients: ["Director of Engineering"]
channel: "Slack DM"
- severity: "SEV1"
timing: "Every 30 minutes"
recipients: ["VP Engineering", "CTO"]
channel: "Email update"
executive_update_template: |
**Incident: [Title]**
Severity: SEV[X]
Status: [Investigating/Mitigating/Resolved]
Impact: [Users affected, services affected]
Duration: [time]
Next update: [time]
**What we know:** [brief description]
**What we're doing:** [current actions]
**What we need:** [escalation requests, if any]Surprising executives with major incidents looks bad. Proactive notification builds trust. Even if you do not have all the details, let them know an incident is happening and you are working on it.
Severity Assignment
The incident commander assigns severity at declaration. Use the impact map and escalation triggers as guidelines. When in doubt, start higher and downgrade later rather than starting low and escalating.
## Severity Assignment Checklist
### Step 1: Assess Impact
- [ ] How many users are affected?
- [ ] What functionality is broken?
- [ ] Is there data loss or security impact?
### Step 2: Map to Severity
- [ ] Check error rate thresholds
- [ ] Check latency thresholds
- [ ] Check availability thresholds
### Step 3: Assign and Communicate
- [ ] Declare severity to team
- [ ] Set up communication cadence
- [ ] Notify stakeholders per policy
### Step 4: Re-evaluate
- [ ] Reassess severity after 15 minutes
- [ ] Escalate or downgrade as needed
- [ ] Update all communication channelsIt is easier to downgrade a SEV1 to SEV2 than to escalate a SEV3 to SEV1 after 30 minutes of delay. When in doubt, start at the higher severity and adjust as you learn more about the impact.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.