Stage 3 · Build
Load Balancing & Traffic Management
Health Checks and Failure Detection
Active vs passive health checks, connection draining, and graceful shutdown.
Active vs Passive Health Checks
| Type | How It Works | Pros | Cons |
|---|---|---|---|
| Active | LB sends probe requests | Fast detection | Extra traffic, false positives |
| Passive | LB monitors real traffic | No extra traffic | Slower detection, needs traffic |
| Hybrid | Both active and passive | Best coverage | More configuration |
Active health checks periodically ping a health endpoint (GET /health). If the endpoint fails or times out, the server is marked unhealthy and removed from the pool. Passive checks observe real traffic — if a server starts returning 5xx errors, it is marked unhealthy.
Active check (every 5 seconds):
LB -> GET /health -> Server
200 OK -> healthy
500 Error or timeout -> unhealthy -> remove from pool
Passive check:
Real request -> Server -> 500 error
Track error rate
Error rate > threshold -> unhealthy -> remove from poolActive checks catch servers that are down but have not received traffic yet. Passive checks catch servers that are responding but returning errors. Use both for comprehensive coverage.
Health Check Configuration
upstream backend {
zone backend 64k;
server 10.0.1.1:8080;
server 10.0.1.2:8080;
server 10.0.1.3:8080;
}
# Commercial Nginx Plus or OpenResty health checks
# Passive checks (monitor real traffic)
server {
location / {
proxy_pass http://backend;
proxy_next_upstream error timeout http_502 http_503;
proxy_next_upstream_tries 3;
}
}proxy_next_upstream tells Nginx to try the next server on specific failure conditions. error means connection error, timeout means slow response, http_502/503 means bad gateway errors. This is passive health checking at the request level.
backend api_servers
balance roundrobin
option httpchk GET /health
http-check expect status 200
server api1 10.0.1.1:8080 check inter 5s fall 3 rise 2
server api2 10.0.1.2:8080 check inter 5s fall 3 rise 2
server api3 10.0.1.3:8080 check inter 5s fall 3 rise 2inter 5s = check every 5 seconds. fall 3 = mark unhealthy after 3 consecutive failures. rise 2 = mark healthy after 2 consecutive successes. These thresholds prevent flapping — servers rapidly toggling between healthy and unhealthy.
Connection Draining
When a server is marked unhealthy, existing connections should be allowed to complete before removing it from the pool. This is connection draining — the server stops receiving new connections but finishes existing ones.
t=0: Server marked unhealthy
- Stop sending NEW connections to this server
- Existing connections continue
t=30s: Draining timeout reached
- Force-close remaining connections
- Server fully removed from pool
Without draining:
- Client requests in flight get reset (RST)
- Users see errors mid-request
With draining:
- In-flight requests complete successfully
- New requests go to healthy serversDraining prevents user-facing errors during deployments and failures. The draining timeout should be long enough for your longest expected request. For most APIs, 30-60 seconds is sufficient.
Circuit Breaking
Circuit breaking prevents cascading failures. If a backend is failing, the circuit breaker trips and immediately returns an error instead of sending more requests to a failing server. After a cooldown, it allows a test request to see if the server recovered.
CLOSED (normal):
Requests flow through
Track failure rate
Failure rate > threshold -> trip -> OPEN
OPEN (blocking):
All requests fail fast (503 or 504)
No traffic hits the failing server
After cooldown -> HALF-OPEN
HALF-OPEN (testing):
Allow one test request through
If it succeeds -> CLOSED
If it fails -> OPEN (reset cooldown)Circuit breaking is critical for microservices. If service B is down, service A should fail fast instead of waiting for timeouts on every request. This prevents thread pool exhaustion and cascading failures across the system.
Your health endpoint must actually verify the service is healthy — not just return 200. Check database connectivity, cache availability, and disk space. A /health endpoint that always returns 200 is worse than no health check at all.
Graceful Shutdown
# 1. Remove from load balancer pool
# (deregister from service discovery or fail health check)
# 2. Wait for connection draining
sleep $DRAIN_TIMEOUT
# 3. Stop accepting new connections
# (server closes listening socket)
# 4. Wait for in-flight requests to complete
# (with a hard timeout)
# 5. Exit
kill -TERM $PID
# Kubernetes example
spec:
terminationGracePeriodSeconds: 60
containers:
- lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 10"]
readinessProbe:
httpGet:
path: /ready
port: 8080
periodSeconds: 5The preStop hook gives Kubernetes time to remove the pod from the Service endpoints before sending SIGTERM. The readiness probe returning false tells the endpoints controller to stop sending traffic. This combination ensures clean shutdown.
Your /ready endpoint should return false during shutdown. This tells the load balancer to stop sending new traffic. The /health endpoint should check dependencies. The /ready endpoint checks if the server is ready to accept traffic.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.