DR Runbooks

## DR Decision Tree

### Step 1: Assess the Situation
- Is the primary region completely down?
  - YES → Go to Step 2
  - NO → Is the outage partial?
    - YES → Can it be resolved within RTO?
      - YES → Attempt resolution
      - NO → Go to Step 2
    - NO → Monitor, no DR needed

### Step 2: Declare DR
- Confirm DR criteria are met:
  - [ ] Primary region unavailable for > 15 minutes
  - [ ] No estimated time to recovery
  - [ ] Business impact is significant
- If criteria met:
  - [ ] Declare DR to leadership
  - [ ] Activate DR team
  - [ ] Open DR war room

### Step 3: Activate Secondary Region
- [ ] Verify secondary region is healthy
- [ ] Promote database replica to primary
- [ ] Update DNS to point to secondary region
- [ ] Verify traffic is flowing to secondary
- [ ] Test key user journeys

### Step 4: Verify and Monitor
- [ ] Verify all services are operational
- [ ] Verify data integrity
- [ ] Monitor error rates and latency
- [ ] Set up ongoing monitoring

### Step 5: Communicate
- [ ] Update status page
- [ ] Notify stakeholders
- [ ] Update leadership

#!/usr/bin/env bash
# DR Runbook: Region Failover Commands

# Step 1: Verify secondary region health
echo "=== Checking secondary region health ==="
kubectl --context=secondary get nodes
kubectl --context=secondary get pods -A | grep -v Running

# Step 2: Promote database replica
echo "=== Promoting database replica ==="
kubectl --context=secondary exec -n database \
  deployment/postgres-primary -- \
  pg_ctl promote

# Step 3: Verify database promotion
echo "=== Verifying database promotion ==="
kubectl --context=secondary exec -n database \
  deployment/postgres-primary -- \
  psql -U postgres -c "SELECT pg_is_in_recovery();"

# Step 4: Update DNS
echo "=== Updating DNS to secondary region ==="
aws route53 change-resource-record-sets \
  --hosted-zone-id Z1234567890 \
  --change-batch file://dns-failover.json

# Step 5: Verify DNS propagation
echo "=== Verifying DNS propagation ==="
dig api.example.com +short
# Expected: secondary region IP

# Step 6: Test application
echo "=== Testing application ==="
curl -s https://api.example.com/healthz
# Expected: {"status":"ok"}

# Step 7: Monitor
echo "=== Monitoring for issues ==="
echo "Check Grafana dashboard: https://grafana.internal/d/dr"
echo "Check error rate: https://grafana.internal/d/errors"

access_prerequisites:
  cloud_provider:
    - "AWS console access to secondary region"
    - "IAM credentials for secondary region"
    - "Route53 access for DNS updates"

  kubernetes:
    - "kubectl context for secondary cluster"
    - "Cluster admin credentials"
    - "Access to all namespaces"

  database:
    - "Database admin credentials for secondary"
    - "SSH access to database servers"
    - "pg_hba.conf update permissions"

  monitoring:
    - "Grafana access for secondary region dashboards"
    - "PagerDuty access for DR alerting"
    - "Log aggregation access"

  communication:
    - "Slack access for DR channel"
    - "Status page admin access"
    - "Email distribution lists"

  verification:
    - "Test access monthly"
    - "Rotate credentials quarterly"
    - "Document access in DR runbook"
    - "Verify access after team changes"

## DR Declaration Notification

**Subject: DISASTER RECOVERY DECLARED — [Service/Region]**

**Severity:** CRITICAL
**Declared by:** [name]
**Time:** [UTC]
**Reason:** [description]

**Current Status:**
- Primary region: [status]
- Secondary region: [status]
- Services affected: [list]

**Next Update:** [time]

---

## DR Status Update

**Subject: DR Update — [Service/Region]**

**Status:** [In Progress / Completed]
**Duration:** [time]
**Current Phase:** [phase]

**Progress:**
- [completed step 1]
- [in progress step 2]
- [pending step 3]

**Next Update:** [time]

---

## DR Resolution Notification

**Subject: DR RESOLVED — [Service/Region]**

**Resolution Time:** [total time]
**Root Cause:** [brief description]
**Impact:** [user impact]

**Post-DR Actions:**
- Failback to primary region: [planned date]
- Postmortem scheduled: [date]
- Action items: [count]

rollback_criteria:
  trigger_rollback:
    - "Secondary region fails health checks"
    - "Data corruption detected during failover"
    - "Error rate exceeds 10% in secondary region"
    - "Critical service unavailable in secondary region"

  rollback_procedure:
    - "Assess if primary region is recovering"
    - "If primary is healthy, initiate failback"
    - "If primary is not healthy, troubleshoot secondary"
    - "Communicate rollback decision to team"

  failback_checklist:
    - "Verify primary region is fully healthy"
    - "Sync data from secondary to primary"
    - "Verify data consistency"
    - "Update DNS to point to primary"
    - "Verify traffic flowing to primary"
    - "Monitor for issues"
    - "Communicate failback completion"

  documentation:
    - "Record rollback reason and timing"
    - "Document any data loss during failover"
    - "Update DR runbook based on lessons learned"