Incident Communications

Slack war rooms, status pages, customer updates, and timestamped decision logs.

6 min readSRE PlaybookOperate

Communication Channels

Every incident needs three communication channels: a war room for coordination, an update channel for status, and a status page for external communication. Mixing channels causes confusion and lost information.

Communication channel structure


communication_channels:
  war_room:
    platform: "Slack"
    name: "#inc-YYYY-MM-DD-brief-description"
    purpose: "Real-time coordination between IC, ops, comms, scribe"
    participants: "Incident roles only"
    rules:
      - "Incident-related messages only"
      - "Use threads for side discussions"
      - "Timestamp important decisions"

  updates:
    platform: "Slack"
    name: "#inc-updates"
    purpose: "Status updates for broader team"
    participants: "All engineering"
    frequency: "Every 15-30 minutes"

  status_page:
    platform: "Statuspage.io"
    purpose: "External customer communication"
    audience: "Public / customers"
    frequency: "Every 30 minutes or on status change"

  executive:
    platform: "Slack DM / Email"
    purpose: "Leadership notification"
    audience: "VP Engineering, CTO"
    frequency: "SEV1 immediately, SEV2 within 30 minutes"

Slack War Room

The war room is the incident's nerve center. Keep it focused, timestamped, and structured. Every message should be either a status update, a decision, or an action item.

War room message format


## War Room Message Format

### Status Update
[HH:MM] STATUS: [Investigating / Identified / Mitigating / Monitoring / Resolved]
- Impact: [description]
- Current state: [what we know]
- Next action: [what we are doing]

### Decision
[HH:MM] DECISION: [decision made]
- Context: [why this decision]
- Owner: [who will execute]
- ETA: [when it will be done]

### Action Item
[HH:MM] ACTION: [what needs to be done]
- Owner: [who]
- Due: [when]
- Status: [pending/in-progress/complete]

### Question
[HH:MM] QUESTION: [question]
- Context: [why asking]
- Who can answer: [targeted question]

### Escalation
[HH:MM] ESCALATION: [what escalated]
- Reason: [why]
- Who: [who is being escalated to]
- ETA: [when]

Status Page Updates

The status page is your source of truth for external communication. Update it on a regular cadence, even if nothing has changed. Silence on the status page breeds uncertainty and support tickets.

Status page update schedule


status_page_updates:
  cadence:
    investigating: "Every 15 minutes"
    identified: "Every 15 minutes"
    monitoring: "Every 30 minutes"
    resolved: "Immediate final update"

  components:
    - name: "API"
      states: ["operational", "degraded_performance", "partial_outage", "major_outage"]
    - name: "Web Dashboard"
      states: ["operational", "degraded_performance", "partial_outage", "major_outage"]
    - name: "Data Pipeline"
      states: ["operational", "degraded_performance", "partial_outage", "major_outage"]

  severity_mapping:
    SEV1: "major_outage"
    SEV2: "partial_outage"
    SEV3: "degraded_performance"
    SEV4: "degraded_performance"

Customer Updates

Customer updates should be clear, concise, and non-technical. Focus on what is affected, what you are doing, and when they can expect resolution. Avoid jargon and internal details.

Customer update templates


## Initial Customer Notification

**Subject: Service Disruption — [Service Name]**

We are experiencing [brief description of issue].
Our engineering team is actively investigating.

**Affected services:** [list]
**Impact:** [user-facing description]
**Started:** [time UTC]

We will provide an update within 30 minutes.

---

## Progress Update

**Subject: Update on Service Disruption — [Service Name]**

We have identified the cause of the disruption.
[Non-technical explanation of what happened.]

**Current status:** [what we are doing]
**Impact:** [current user impact]
**Next update:** [time]

---

## Resolution Notice

**Subject: Service Restored — [Service Name]**

The issue affecting [service name] has been resolved.

**Duration:** [total time]
**Root cause:** [non-technical explanation]
**Resolution:** [what we did to fix it]

A detailed post-incident review will be published within 5 business days.

We apologize for the inconvenience and are taking steps to prevent recurrence.

Decision Logging

Log every significant decision during an incident. This creates an audit trail for the postmortem and helps future incident commanders understand why certain choices were made.

Decision log format


decision_log:
  - timestamp: "2024-01-15T14:30:00Z"
    decision: "Declare SEV1 incident"
    rationale: "Error rate > 50% for 2 minutes, all users affected"
    made_by: "IC: Alice"
    alternatives_considered:
      - "Wait and monitor (rejected: impact too high)"

  - timestamp: "2024-01-15T14:35:00Z"
    decision: "Rollback deployment v1.2.3 to v1.2.2"
    rationale: "Recent deploy correlates with error spike"
    made_by: "IC: Alice, with input from Ops: Bob"
    alternatives_considered:
      - "Scale up (rejected: does not address root cause)"
      - "Restart pods (rejected: may not help if deploy is cause)"

  - timestamp: "2024-01-15T14:45:00Z"
    decision: "Notify VP Engineering"
    rationale: "SEV1 exceeds executive notification threshold"
    made_by: "IC: Alice"
    alternatives_considered: []

Use threads for decisions

In Slack, use threads to discuss decisions. The top-level message is the decision itself. Replies contain the discussion, alternatives, and rationale. This keeps the main channel clean while preserving context.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 6 · Operate

Incident Response & On-Call