Stage 5 · Platform
Networking & Service Mesh
Traffic Shaping
Retries, timeouts, circuit breakers, fault injection, and weighted canary routes.
Retry Policies
Retries handle transient failures — temporary network glitches, pod restarts, or overloaded backends. Configure retries with limits, timeouts, and retry-on conditions. Poor retry configuration can amplify failures (retry storms).
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: api-service
spec:
hosts:
- api-service
http:
- route:
- destination:
host: api-service
retries:
attempts: 3
perTryTimeout: 2s
retryOn: "5xx,reset,connect-failure,retriable-4xx"
retryRemoteLocalities: true
timeout: 10sattempts: 3 means up to 3 retries (plus the original request). perTryTimeout: 2s limits each attempt to 2 seconds. retryOn specifies which errors trigger retries. total timeout is 10s for all attempts combined.
If 10% of requests fail and you retry 3 times, you generate 30% more load. In a service chain with 5 hops, retry amplification can be 3x. Use retry budgets (% of total requests) instead of fixed attempts to prevent retry storms.
Timeouts
Timeouts prevent requests from hanging indefinitely. Every service-to-service call should have a timeout. Without timeouts, slow backends cause cascading failures — threads pile up waiting, the caller runs out of capacity, and the entire system degrades.
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: api-service
spec:
hosts:
- api-service
http:
- route:
- destination:
host: api-service
timeout: 5s
---
# DestinationRule connection pool limits
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: api-service
spec:
host: api-service
trafficPolicy:
connectionPool:
tcp:
maxConnections: 100
connectTimeout: 3s
http:
h2UpgradePolicy: DEFAULT
maxRequestsPerConnection: 10
maxRequests: 100
maxRetries: 3timeout: 5s on VirtualService limits the total request time. connectTimeout: 3s on DestinationRule limits the TCP connection establishment. maxRequests limits concurrent HTTP requests to the backend.
Circuit Breakers
Circuit breakers prevent cascading failures by stopping requests to failing backends. When a backend fails, the circuit opens and requests fail fast. After a cooldown, the circuit half-opens to test if the backend has recovered.
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: api-service
spec:
host: api-service
trafficPolicy:
connectionPool:
tcp:
maxConnections: 100
http:
maxRequests: 50
maxPendingRequests: 25
maxRetries: 3
outlierDetection:
consecutive5xxErrors: 5
interval: 10s
baseEjectionTime: 30s
maxEjectionPercent: 50
minHealthPercent: 30outlierDetection implements the circuit breaker: after 5 consecutive 5xx errors within 10s, the endpoint is ejected for 30s. maxEjectionPercent: 50 prevents ejecting all endpoints. minHealthPercent: 30 stops ejecting when health drops below 30%.
Fault Injection
Fault injection deliberately introduces failures to test resilience. You can inject HTTP errors (503), delays (latency), and abort requests. This is chaos engineering for service meshes — test how your system handles failures before they happen in production.
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: api-service
spec:
hosts:
- api-service
http:
- fault:
delay:
percentage:
value: 10 # 10% of requests
fixedDelay: 5s # Add 5 second delay
abort:
percentage:
value: 5 # 5% of requests
httpStatus: 503 # Return 503
route:
- destination:
host: api-serviceThis injects a 5s delay into 10% of requests and aborts 5% of requests with 503. Use this to test retry logic, timeout handling, and circuit breaker behavior. Remove fault injection after testing.
Weighted Canary Routes
Weighted routes split traffic between versions. Start with a small percentage on the canary (1-5%), monitor for errors, and gradually increase. This is the foundation of canary deployments and progressive delivery.
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: api-canary
spec:
hosts:
- api-service
http:
- route:
- destination:
host: api-service
subset: stable
weight: 95
- destination:
host: api-service
subset: canary
weight: 5
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: api-destinations
spec:
host: api-service
subsets:
- name: stable
labels:
version: v1
- name: canary
labels:
version: v2This routes 95% to stable (v1) and 5% to canary (v2). Monitor error rates, latency, and business metrics. If the canary looks healthy, increase weight gradually: 5% -> 25% -> 50% -> 100%.
Resilience Patterns
- Retry with budget — Limit retries to % of total requests, not fixed count
- Timeout cascade — Set timeouts that decrease as you go deeper in the call chain
- Circuit breaker — Open circuit after consecutive failures, half-open to test recovery
- Bulkhead — Isolate failure domains with connection pool limits per service
- Rate limiting — Protect backends from overload with client-side rate limits
Use Istio fault injection to test retries, timeouts, and circuit breakers. Use Chaos Mesh or Litmus Chaos to inject pod failures, network partitions, and resource pressure. Test regularly — resilience degrades without practice.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.