Stage 4 · Provision
Resilience Patterns
Failure Injection Tests
Validating resilience with Toxiproxy, Chaos Mesh, latency injection, and dependency blackholes.
Why Inject Failures?
Failure injection validates that your resilience patterns actually work. Circuit breakers, retries, and fallbacks are theoretical until tested under real failure conditions. Injecting failures in a controlled environment proves your system handles them correctly.
Toxiproxy
Toxiproxy is a TCP proxy for simulating network conditions. It sits between your service and its dependencies, injecting latency, packet loss, connection resets, and bandwidth limits. It is lightweight, easy to integrate, and perfect for integration tests.
listen: "0.0.0.0:8474"
upstreams:
- name: "postgres"
listen: "0.0.0.0:5433"
upstream: "postgres:5432"
toxics:
- name: "latency"
type: "latency"
attributes:
latency: 1000 # 1 second latency
- name: "timeout"
type: "timeout"
attributes:
timeout: 5000 # 5 second timeout
- name: "redis"
listen: "0.0.0.0:6380"
upstream: "redis:6379"
toxics:
- name: "slow_close"
type: "slow_close"
attributes:
duration: 500 # 500ms delay on closeToxiproxy intercepts connections and applies toxic effects. Your application connects to Toxiproxy instead of the real dependency.
Chaos Mesh
Chaos Mesh is a Kubernetes-native chaos engineering platform. It injects Pod failures, network partitions, DNS disruptions, IO latency, and time skew. Chaos Mesh runs as a Kubernetes operator and provides a dashboard for managing experiments.
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
name: "payment-network-partition"
spec:
action: partition
mode: all
selector:
labelSelectors:
app: payment-service
direction: both
target:
selector:
labelSelectors:
app: payment-gateway
mode: all
duration: "5m"This experiment creates a network partition between payment-service and payment-gateway for 5 minutes. The system should detect the failure and activate circuit breakers.
Latency Injection
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
name: "database-latency"
spec:
action: delay
mode: all
selector:
labelSelectors:
app: postgres
delay:
latency: "500ms"
jitter: "100ms"
correlation: "50"
duration: "10m"
direction: toThis adds 500ms latency (with 100ms jitter) to all traffic going to PostgreSQL for 10 minutes. The application should activate circuit breakers and serve cached data.
Dependency Blackholes
A blackhole drops all traffic to a dependency without responding. This simulates a complete dependency failure. The service must detect the blackhole, activate circuit breakers, and serve fallback responses within the timeout budget.
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
name: "redis-blackhole"
spec:
action: loss
mode: all
selector:
labelSelectors:
app: redis
loss:
loss: "100" # 100% packet loss
correlation: "100"
duration: "3m"A 100% packet loss simulates a complete Redis failure. The application should detect the failure, activate circuit breakers, and serve responses from local cache.
Automated Resilience Testing
- Define hypothesis — What should happen when X fails?
- Inject failure — Use Toxiproxy or Chaos Mesh.
- Observe behavior — Check metrics, logs, and traces.
- Validate hypothesis — Did the circuit breaker open? Did fallback serve?
- Automate — Run resilience tests in CI/CD pipeline.
Network latency is the safest failure to inject first. It simulates real-world network conditions without causing data loss. Start with 100ms latency and increase to 5s. This validates timeout and circuit breaker behavior.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.