Stage 6 · Operate
Reliability Testing & Chaos Engineering
Experiment Design
Hypotheses, steady-state metrics, blast radius controls, abort criteria, and observer roles.
Experiment Principles
Reliability experiments test your system's ability to handle failure. They follow the scientific method: form a hypothesis, design an experiment, observe results, and draw conclusions. The goal is not to break things — it is to learn how your system behaves under stress.
A reliability experiment is not about causing chaos for its own sake. It is about understanding your system's behavior, validating assumptions, and identifying weaknesses before they cause real incidents.
Forming Hypotheses
hypothesis:
template: "If we inject [failure], then [expected behavior] because [reasoning]"
examples:
- hypothesis: "If we kill a database pod, then the application
will continue serving requests because the replica will
be promoted within 30 seconds"
validation: "Error rate remains below 1% during failover"
- hypothesis: "If we increase latency to a downstream service
by 500ms, then the circuit breaker will open and the
application will serve cached responses"
validation: "Circuit breaker opens within 10 seconds, cached
responses served, error rate stays below 0.5%"
- hypothesis: "If we fill the disk to 95%, then the log rotation
will trigger and free space before the application crashes"
validation: "Disk usage drops below 80% within 5 minutes"
criteria:
- "Specific: clearly states what failure is injected"
- "Measurable: defines expected behavior quantitatively"
- "Achievable: experiment can be conducted safely"
- "Relevant: tests a real failure mode"
- "Time-bound: has clear start and end criteria"Steady-State Metrics
Before injecting failure, establish what normal looks like. Steady-state metrics are your baseline. During the experiment, compare actual behavior to steady-state to determine if the system is healthy.
steady_state:
purpose: "Define normal behavior before experiment"
metrics:
availability:
metric: "Error rate < 0.1%"
measurement: "rate(http_request_duration_seconds_count{code=~'5..'}[5m]) / rate(http_request_duration_seconds_count[5m])"
threshold: 0.001
latency:
metric: "P99 latency < 200ms"
measurement: "histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))"
threshold: 0.2
throughput:
metric: "Request rate within 10% of baseline"
measurement: "rate(http_request_duration_seconds_count[5m])"
baseline: "measured during 5-minute observation"
saturation:
metric: "CPU < 70%, Memory < 80%"
measurement: "avg(rate(container_cpu_usage_seconds_total[5m]))"
observation_period: "5 minutes before experiment"
confirmation: "All metrics within thresholds for observation period"Blast Radius Controls
Control the scope of your experiment. Start small, validate results, then expand. Never inject failure into a system you cannot roll back quickly.
blast_radius:
levels:
level_1:
description: "Single pod in staging"
scope: "1 pod"
duration: "5 minutes"
monitoring: "Continuous"
approval: "Experiment owner"
level_2:
description: "Multiple pods in staging"
scope: "25% of pods"
duration: "10 minutes"
monitoring: "Continuous"
approval: "Experiment owner + SRE lead"
level_3:
description: "Full staging environment"
scope: "All pods"
duration: "15 minutes"
monitoring: "Continuous"
approval: "SRE lead"
level_4:
description: "Single pod in production (low-traffic)"
scope: "1 pod, low-traffic service"
duration: "5 minutes"
monitoring: "Continuous"
approval: "SRE lead + VP Engineering"
controls:
- "Start at level 1, progress only if successful"
- "Have rollback command ready before starting"
- "Monitor continuously during experiment"
- "Abort if steady-state metrics are exceeded"
- "Document everything for post-experiment review"Abort Criteria
Define clear abort criteria before starting the experiment. If any criteria are met, stop immediately. Abort criteria prevent experiments from becoming incidents.
abort_criteria:
immediate_abort:
- "Error rate exceeds 5% for any 1-minute window"
- "Latency p99 exceeds 2s for any 1-minute window"
- "Availability drops below 99% for any 5-minute window"
- "User reports impact"
- "Downstream service reports impact"
- "Data corruption detected"
escalation_abort:
- "Error rate exceeds 2% for 5 minutes"
- "Latency p99 exceeds 1s for 5 minutes"
- "Experiment duration exceeds planned time by 50%"
abort_procedure:
- "Immediately stop failure injection"
- "Verify system recovers to steady state"
- "If no recovery, execute rollback"
- "Document abort reason and conditions"
- "Conduct debrief within 24 hours"
post_abort:
- "Do not retry the same experiment"
- "Analyze why abort was triggered"
- "Fix underlying issue before retesting"
- "Update abort criteria if needed"Observer Role
The observer watches the experiment without participating. They monitor metrics, track timeline, and make the abort decision. The observer is the safety net.
observer_role:
responsibilities:
- "Monitor steady-state metrics continuously"
- "Track experiment timeline"
- "Make abort decisions based on criteria"
- "Document observations in real-time"
- "Ensure rollback is available"
- "Coordinate with team if escalation needed"
before_experiment:
- "Verify steady-state metrics are normal"
- "Confirm rollback command is ready"
- "Set up monitoring dashboards"
- "Verify abort criteria are clear"
- "Confirm all participants understand their roles"
during_experiment:
- "Monitor metrics continuously"
- "Call out metric deviations"
- "Make abort decision if criteria are met"
- "Document observations with timestamps"
after_experiment:
- "Verify system returned to steady state"
- "Document experiment results"
- "Identify unexpected behaviors"
- "Recommend follow-up experiments"
anti_patterns:
- "Getting pulled into debugging"
- "Ignoring metric deviations"
- "Being afraid to abort"
- "Not documenting observations"Your first reliability experiment should be simple: kill a pod and verify it restarts. This validates your experiment process, tools, and abort criteria before you attempt more complex experiments.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.