Incident Response for Platform Teams

# Runbook: Platform API High Latency

## Metadata
- **Component:** Platform API
- **Owner:** @platform-oncall
- **Severity:** Critical (blocks all deploys)
- **Last Updated:** 2024-01-15
- **Review Cycle:** Quarterly

## Symptoms
- Platform API p99 latency > 1s for 5 minutes
- Scaffold commands timing out
- ArgoCD ApplicationSet controller errors
- Multiple teams reporting "platform slow"

## Detection
- **Alert:** PlatformAPIHighLatency (PrometheusRule)
- **Query:** histogram_quantile(0.99, rate(platform_api_request_duration_seconds_bucket[5m])) > 1
- **Dashboard:** Grafana > Platform > API Latency

## Diagnosis Steps
1. **Check API pods:**
   ```bash
   kubectl get pods -n platform -l app=platform-api
   kubectl top pods -n platform -l app=platform-api
   ```
   - Look for: OOMKilled, CPU throttling, restart loops

2. **Check dependencies:**
   - PostgreSQL: `kubectl get pods -n platform -l app=postgres`
   - Redis: `kubectl get pods -n platform -l app=redis`
   - GitHub API: Check GitHub status page

3. **Check recent changes:**
   - ArgoCD: `argocd app list | grep platform-api`
   - GitHub: Recent PRs to platform-api repo
   - ConfigMap changes: `kubectl get events -n platform --sort-by=.metadata.creationTimestamp`

4. **Check resource saturation:**
   - Node CPU/memory: `kubectl top nodes`
   - API server latency: `kubectl get --raw /metrics | grep apiserver_request_duration`

## Mitigation
### Immediate (5 min)
- [ ] Scale API replicas: `kubectl scale deployment platform-api -n platform --replicas=10`
- [ ] Enable read-only mode (if feature flag exists): `kubectl set env deployment/platform-api -n platform READ_ONLY=true`
- [ ] Communicate: Post in #platform-incidents, update statuspage.io

### Short-term (15 min)
- [ ] Identify root cause (DB, Redis, CPU, memory, downstream)
- [ ] Apply targeted fix (scale DB, clear Redis, rollback deploy)
- [ ] Verify latency returning to normal

### Long-term (1 hr)
- [ ] Full root cause analysis
- [ ] Preventive measures (autoscaling, circuit breakers, caching)
- [ ] Update runbook with learnings

## Contacts
- **Primary On-Call:** @platform-oncall (PagerDuty)
- **Secondary:** @platform-lead
- **DB Admin:** @dba-oncall
- **Infra Lead:** @infra-lead
- **Comms:** @platform-comms

## Escalation
- **15 min:** No progress → Page secondary + infra lead
- **30 min:** No progress → Page platform director + CTO
- **60 min:** No progress → All-hands war room

## Post-Incident
- [ ] Create incident record in incident.io
- [ ] Schedule postmortem within 48 hours
- [ ] Update runbook with new learnings
- [ ] Add preventive alerts if missing

# Incident Timeline: INC-20240115-001

## Summary
- **Incident ID:** INC-20240115-001
- **Title:** Platform API High Latency (p99 > 5s)
- **Severity:** SEV-1 (Critical)
- **Start:** 2024-01-15 10:23 UTC
- **End:** 2024-01-15 11:45 UTC
- **Duration:** 82 minutes
- **Commander:** @jane.platform
- **Scribe:** @bob.platform

## Timeline
| Time (UTC) | Event | Action | Owner |
|------------|-------|--------|-------|
| 10:23 | Alert fired: PlatformAPIHighLatency | PagerDuty alert to @platform-oncall | PagerDuty |
| 10:25 | On-call acknowledges | Joined #inc-20240115-platform-api | @jane.platform |
| 10:26 | War room created | #inc-20240115-platform-api | @jane.platform |
| 10:28 | Diagnosis started | Checked API pods, DB, Redis | @jane.platform |
| 10:32 | Root cause identified | PostgreSQL CPU 100% due to missing index | @jane.platform |
| 10:35 | Mitigation: Scale read replicas | `kubectl scale statefulset postgres-read -n platform --replicas=5` | @bob.platform |
| 10:40 | Latency improving | p99 dropped from 5s → 800ms | @jane.platform |
| 10:45 | Index added | `CREATE INDEX CONCURRENTLY...` | @dba.oncall |
| 10:50 | Latency normal | p99 < 200ms | @jane.platform |
| 10:55 | Monitoring confirmed | 15 min stable | @jane.platform |
| 11:00 | Incident closed | Status page updated, team notified | @jane.platform |

## Root Cause
Missing index on `audit_logs.service_name` caused sequential scan on 50M row table during scaffold audit logging.

## Action Items
- [ ] Add index to audit_logs table (DBA) - DONE
- [ ] Add query plan analysis to CI for audit logging queries (Platform) - IN PROGRESS
- [ ] Add alert for PostgreSQL CPU > 80% (Platform) - TODO
- [ ] Review all audit logging queries for indexes (Platform) - IN PROGRESS

## Communication Log
- 10:30: Statuspage.io updated: "Investigating platform API latency"
- 10:45: Statuspage.io updated: "Identified root cause, implementing fix"
- 11:00: Statuspage.io updated: "Resolved - Platform API latency restored"
- 11:05: #platform-updates: "Platform API latency incident resolved. Root cause: missing DB index. Postmortem scheduled."
- 11:15: Email to stakeholders sent

# Postmortem: INC-20240115-001

## Overview
- **Incident ID:** INC-20240115-001
- **Title:** Platform API High Latency due to Missing DB Index
- **Date:** 2024-01-15
- **Duration:** 82 minutes (10:23 - 11:45 UTC)
- **Severity:** SEV-1
- **Author:** @jane.platform
- **Reviewers:** @platform-lead, @dba-lead, @infra-lead

## Impact
- **Services Affected:** Platform API, Scaffold, ArgoCD ApplicationSet, Crossplane
- **Teams Affected:** All 12 teams (blocked deploys, scaffold, preview envs)
- **Users Affected:** ~150 developers
- **Revenue Impact:** None (internal platform)
- **SLA Breach:** Platform API p99 latency SLO (99.9% < 500ms) breached for 82 min

## Timeline
[Link to timeline document]

## Root Cause
**Primary:** Missing index on `audit_logs.service_name` caused sequential scan on 50M row table during scaffold audit logging.

**Contributing Factors:**
1. New audit logging feature deployed without query plan review
2. No automated query plan analysis in CI
3. No alert on PostgreSQL CPU saturation
4. Audit logging writes synchronous in scaffold path

## What Went Well
- Alert fired within 2 minutes of latency spike
- On-call responded within 3 minutes
- War room established quickly
- Root cause identified in 9 minutes
- DBA available and responsive
- Fix applied and verified within 27 minutes of root cause

## What Went Poorly
- No automated query plan analysis in CI/CD
- No alert on PostgreSQL CPU saturation
- Audit logging synchronous in critical path
- No circuit breaker for audit logging failures
- Status page update delayed by 7 minutes

## Root Cause Analysis (5 Whys)
1. **Why high latency?** PostgreSQL CPU 100%
2. **Why CPU 100%?** Sequential scan on audit_logs table
3. **Why sequential scan?** Missing index on service_name column
4. **Why missing index?** Query plan not reviewed during audit logging feature development
5. **Why no review?** No automated query plan check in CI; manual review skipped due to time pressure

## Action Items
| # | Action | Owner | Due Date | Status | Tracking |
|---|--------|-------|----------|--------|----------|
| 1 | Add missing index on audit_logs.service_name | @dba.oncall | 2024-01-15 | ✅ Done | PR #1234 |
| 2 | Add query plan analysis to CI for all DB migrations | @platform-lead | 2024-02-01 | 🚧 In Progress | Issue #5678 |
| 3 | Add PostgreSQL CPU > 80% alert | @platform-oncall | 2024-01-22 | 📋 Planned | Issue #5679 |
| 4 | Make audit logging async with circuit breaker | @platform-lead | 2024-02-15 | 📋 Planned | Issue #5680 |
| 5 | Add query plan review to code review checklist | @platform-lead | 2024-01-22 | 📋 Planned | Issue #5681 |
| 6 | Review all existing audit logging queries for indexes | @dba-lead | 2024-02-01 | 📋 Planned | Issue #5682 |

## Lessons Learned
1. **Automate what humans forget:** Query plan analysis must be automated in CI
2. **Async by default:** Critical path operations should not block on observability
3. **Alert on symptoms, not just causes:** CPU saturation alert would have caught this earlier
4. **Invest in tooling:** Automated query analysis pays for itself in prevented incidents

## Appendix
- [Incident Timeline](link)
- [Grafana Dashboard Snapshot](link)
- [PostgreSQL Query Plans](link)
- [Related Incidents](link)