Stage 4 · Provision
Designing for High Availability
Chaos Engineering
Hypothesis-driven failure injection to validate resilience before production incidents expose weaknesses.
What Is Chaos Engineering?
Chaos engineering is the discipline of experimenting on a system to build confidence in its ability to withstand turbulent conditions in production. Rather than waiting for failures to happen, you intentionally inject failures to discover weaknesses before they cause outages.
Principles of Chaos
- Start with a steady state hypothesis — Define what normal behavior looks like.
- Introduce realistic failures — Kill processes, inject latency, fill disks, partition networks.
- Observe the difference — Measure the impact on steady-state behavior.
- Validate the hypothesis — Did the system behave as expected?
- Fix the weakness — If the hypothesis was wrong, fix the system.
Chaos engineering is not about randomly breaking things. It is about forming hypotheses and testing them systematically. Every experiment has a clear steady state, a failure injection, and expected outcomes.
Designing Experiments
experiment:
name: "api-latency-injection"
hypothesis: "API should return cached results when database is slow"
steady_state:
metric: "api_error_rate"
threshold: "< 0.1%"
attack:
type: "latency_injection"
target: "database-connection"
latency_ms: 5000
duration: "5m"
rollback:
auto: true
trigger: "error_rate > 5%"
success_criteria:
- "api_error_rate remains < 1%"
- "p99_latency < 2s"
- "cache_hit_rate increases"A well-structured experiment defines what normal looks like, what failure you inject, and what success means. The rollback trigger ensures the experiment does not cause an actual outage.
Chaos Tools
| Tool | Type | Use Case | Scale |
|---|---|---|---|
| Chaos Monkey | Instance termination | Random VM/container kills | AWS |
| Gremlin | Full platform | CPU, memory, network, process attacks | Kubernetes, VMs |
| Litmus | Kubernetes-native | Pod/node failures, network partitions | Kubernetes |
| Chaos Mesh | Kubernetes-native | Network, IO, kernel, time attacks | Kubernetes |
| Toxiproxy | Network proxy | Simulate network failures between services | Docker, K8s |
Game Days
Game days are coordinated chaos exercises where the entire team responds to a simulated incident. They test not just technical resilience but also human processes: communication, escalation, incident response, and recovery procedures.
- Announce the game day in advance — surprise game days cause panic.
- Assign roles: incident commander, communicator, technical lead.
- Inject failures progressively, not all at once.
- Debrief afterward: what worked, what broke, what needs fixing.
- Track improvement over time with documented action items.
Safety Boundaries
Chaos experiments must have clear safety boundaries. Never inject failures into production without automated rollback mechanisms, blast radius limits, and the ability to abort instantly. Start in staging, graduate to production with guardrails.
Always have automated kill switches, blast radius limits, and rollback mechanisms. Start with read-only experiments (latency injection, DNS changes) before moving to destructive ones (process kills, disk fills).
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.