Stage 3 · Build
Network Debugging in Production
Incident Practice: Network Outage
Step through a realistic production network incident from alert to resolution.
The Scenario
It is 3 AM. PagerDuty alerts fire. The monitoring dashboard shows 5xx error rates spiking to 40%. Users cannot load the main dashboard. The API gateway is returning 502 Bad Gateway for 30% of requests. Your job: diagnose and resolve the issue.
Alert: 5xx error rate > 20% for 5 minutes
Dashboard: API gateway returning 502 for dashboard endpoints
Scope: All users, intermittent (not all requests fail)
Duration: Started 10 minutes ago
Recent changes: Deployment 2 hours ago (v2.3.1 of user-service)502 Bad Gateway means the load balancer or API gateway received an invalid response from the backend. This is different from 503 (service unavailable) — the backend is responding, but something is wrong with the response.
Initial Response
# Check overall error rate
curl -s "http://prometheus:9090/api/v1/query?query=rate(http_requests_total{code=~'5..'}[5m])" | jq
# Check which services are affected
kubectl get pods -A | grep -v Running
# Check recent deployments
kubectl rollout history deployment/user-service
# Check pod restarts (crashes)
kubectl get pods -A --field-selector=status.phase!=RunningFirst: scope the problem. Is it all services or just one? Are pods crashing? Was there a recent deployment? The deployment 2 hours ago is suspicious but not necessarily the cause.
# API gateway logs (nginx/ingress)
kubectl -n ingress-nginx logs -l app.kubernetes.io/name=ingress-nginx --tail=100 | grep "502"
# User-service logs
kubectl logs -l app=user-service --tail=50
# Check endpoint health
kubectl get endpoints user-service
# Check pod readiness
kubectl describe pod -l app=user-service | grep -A 5 "Conditions:"502 errors from the ingress controller mean the backend returned something unexpected. Check: is the backend returning a valid HTTP response? Is the connection being reset? Is the backend timing out?
Isolation Phase
# Test the user-service directly (bypass load balancer)
kubectl exec -it debug-pod -- curl -v http://user-service:8080/health
# Check if the issue is specific to certain pods
for pod in $(kubectl get pods -l app=user-service -o name); do
echo "Testing $pod..."
kubectl exec -it $pod -- curl -s http://localhost:8080/health
done
# Check network between ingress and user-service
kubectl exec -it debug-pod -- traceroute user-service
# Check DNS resolution
kubectl exec -it debug-pod -- dig user-service.default.svc.cluster.local +shortTest each component in isolation. If direct requests to the pod work but through the service they fail, the issue is with kube-proxy or the service. If some pods work and others do not, the issue is pod-specific (bad deployment, resource exhaustion).
Finding the Root Cause
# Found: user-service pods are returning malformed responses
# for /api/users endpoint (50% of the time)
# Check the application logs for errors
kubectl logs -l app=user-service --tail=200 | grep -i error
# Found: database connection pool exhausted
# "Error: Cannot acquire connection from pool"
# Check database health
kubectl exec -it debug-pod -- pg_isready -h postgres-service
# Check connection count to database
kubectl exec -it debug-pod -- ss -tn dst postgres-service:5432 | wc -l
# Check if the recent deployment changed connection pool settings
kubectl describe deployment user-service | grep -A 10 "Environment:"The root cause chain: deployment increased connection pool size -> database max_connections exceeded -> connections fail -> some requests return errors -> ingress sees malformed responses -> 502 errors. The deployment 2 hours ago is the cause.
Resolution and Post-Mortem
# Option A: Roll back the deployment
kubectl rollout undo deployment/user-service
# Option B: Fix the configuration
kubectl set env deployment/user-service DB_POOL_SIZE=20
# Wait for rollout to complete
kubectl rollout status deployment/user-service
# Verify the fix
curl -v https://api.example.com/health
# Confirm error rate is dropping
curl -s "http://prometheus:9090/api/v1/query?query=rate(http_requests_total{code=~'5..'}[5m])" | jqRoll back first, fix later. In an incident, the priority is restoring service. Once the incident is resolved, do a post-mortem to understand why the deployment was not caught in staging.
## Incident: user-service 502 errors
### Timeline
- T+0: Deployment v2.3.1 rolled out
- T+2h: Database connection pool exhaustion begins
- T+2h10m: PagerDuty alert fires
- T+2h15m: On-call engineer begins investigation
- T+2h35m: Root cause identified (DB pool size)
- T+2h40m: Rollback deployed
- T+2h50m: Error rate returns to normal
### Root Cause
Deployment increased DB connection pool from 10 to 50,
exceeding PostgreSQL max_connections (100) when
all 20 pods connected simultaneously.
### Action Items
1. Add max_connections monitoring
2. Add connection pool limits to deployment review
3. Test database impact in staging
4. Add connection pool health check to /health endpointPost-mortems are blameless. Focus on the process, not the person. The goal is to prevent recurrence, not assign blame. Every incident should produce actionable items that make the system more resilient.
1. Communicate early (update status page). 2. Mitigate first (rollback, feature flag, scale). 3. Diagnose after (root cause analysis). 4. Learn from it (post-mortem). Never skip the post-mortem — that is how teams get better.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.