Stage 6 · Operate

Incident Response & RCA

Debugging Under Pressure

Structured debugging, hypothesis-driven investigation, and staying calm when it matters most.

7 min readSRE PlaybookOperate

Debugging Mindset

Production debugging under pressure is different from debugging in development. You cannot reproduce issues at will. You cannot take your time. You need a structured approach that maximizes information gathering while minimizing risk.

Stay calm, stay systematic

The instinct during an incident is to try random fixes. Resist this. A systematic approach — form hypothesis, gather evidence, test hypothesis, act on evidence — is faster and safer than random changes.

Hypothesis-Driven Investigation

Start with a hypothesis based on the symptoms. A hypothesis is a testable statement that, if true, explains the observed behavior. Form hypotheses from the most likely causes, test them in order, and eliminate them systematically.

Hypothesis-driven debugging example


symptom: "API returning 503 errors for 10% of requests"

hypothesis_1:
  statement: "Pods are OOMing"
  test: "kubectl get pods -o json | jq '.items[] | select(.status.containerStatuses[].lastState.terminated.reason=="OOMKilled")'"
  expected: "OOMKilled pods found"
  result: "No OOMKilled pods"
  conclusion: "Hypothesis rejected"

hypothesis_2:
  statement: "Database connection pool exhausted"
  test: "curl -s http://user-api:8080/debug/vars | jq '.db_pool_active, .db_pool_idle'"
  expected: "Active connections at max, idle at 0"
  result: "Active: 45/50, idle: 5"
  conclusion: "Hypothesis confirmed — connection pool nearly exhausted"

action: "Scale database connection pool from 50 to 100"

Debugging Checklist

Keep a debugging checklist for common failure modes. This prevents you from missing obvious causes during high-stress situations. Update the checklist after every incident.

Production debugging checklist


## Quick Checks (First 5 minutes)
- [ ] Check recent deployments: kubectl rollout history
- [ ] Check pod status: kubectl get pods -o wide
- [ ] Check events: kubectl get events --sort-by=.lastTimestamp
- [ ] Check resource usage: kubectl top pods
- [ ] Check error logs: kubectl logs -l app=SERVICE --tail=100

## Infrastructure (Next 10 minutes)
- [ ] Node health: kubectl get nodes
- [ ] Disk space: df -h on relevant nodes
- [ ] Network connectivity: ping/curl between services
- [ ] DNS resolution: nslookup service-name
- [ ] Certificate expiry: openssl s_client -connect host:443

## Application (Next 15 minutes)
- [ ] Application logs: grep for ERROR/WARN/FATAL
- [ ] Metrics dashboard: latency, error rate, saturation
- [ ] Dependency health: downstream service status
- [ ] Configuration changes: recent ConfigMap/Secret updates
- [ ] Database state: connection count, slow queries, locks

## Escalation Triggers
- [ ] No progress after 15 minutes
- [ ] Root cause uncertain after 30 minutes
- [ ] Impact expanding despite mitigation
- [ ] Requires access or knowledge you do not have

Common Anti-Patterns

During incidents, engineers fall into predictable traps. Recognizing these anti-patterns helps you avoid them.

Blame game — Searching for who caused the problem instead of what caused it.
Fix first, understand later — Applying fixes without understanding the root cause.
Tunnel vision — Focusing on one hypothesis and ignoring contradictory evidence.
Too many cooks — Multiple people editing the same resources without coordination.
Hero syndrome — One person doing everything instead of delegating roles.
Skip the timeline — Not documenting events as they happen.

Avoid hero culture

Hero culture is dangerous. If only one person can debug the system, you have a single point of failure. Build shared debugging skills through game days and knowledge sharing. No one should be irreplaceable.

Tools and Techniques

Have your debugging tools ready before an incident. Bookmark Grafana dashboards, prepare kubectl shortcuts, and keep a terminal with common commands ready. Speed matters when debugging production.

Useful production debugging commands


# Check pod resource usage
kubectl top pods -n production --sort-by=memory

# Tail logs with timestamp and context
kubectl logs -l app=user-api -n production --tail=50 --prefix --timestamps

# Check recent events
kubectl get events -n production --sort-by=.lastTimestamp | tail -20

# Execute into a pod for debugging
kubectl exec -it deployment/user-api -n production -- /bin/sh

# Check network connectivity between pods
kubectl run debug --image=curlimages/curl -it --rm -- \
  curl -s http://user-api.production.svc:8080/healthz

# Port-forward for local debugging
kubectl port-forward svc/user-api 8080:8080 -n production

# Check certificate expiry
echo | openssl s_client -connect api.example.com:443 2>/dev/null | \
  openssl x509 -noout -dates

When to Escalate

Escalate when you are stuck, when the impact is growing, or when you need access or expertise you do not have. Escalation is not failure — it is following the process.

Situation	Action
No progress after 15 minutes	Escalate to SRE lead
Root cause uncertain after 30 minutes	Escalate to senior SRE
Impact expanding despite mitigation	Escalate to IC and manager
Need database access	Escalate to DBA team
Need network debugging	Escalate to network team
Customer data affected	Escalate to security team

Use the 15-minute rule

If you have not made progress in 15 minutes, escalate. This is not about ability — it is about speed. Fresh eyes and additional expertise resolve incidents faster than stubbornness.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.