Stage 4 · Provision
Resilience Patterns
Circuit Breakers
Open, half-open, and closed states with failure thresholds in Hystrix and Envoy.
Why Circuit Breakers?
A circuit breaker stops calling a failing dependency after a threshold of failures. This gives the dependency time to recover and prevents cascading failures. Without circuit breakers, retries against a dead service create a thundering herd that makes recovery impossible.
The Three States
Success
┌───────────────┐
│ │
▼ │
┌─────────┐ Failures ┌─────────┐
│ CLOSED │──threshold──►│ OPEN │
└─────────┘ └─────────┘
▲ │
│ │ timeout
│ ▼
│ ┌─────────┐
└──success──────────│HALF-OPEN│
└─────────┘
│ failure
▼
┌─────────┐
│ OPEN │
└─────────┘CLOSED: Normal operation, requests pass through. OPEN: Failing, requests are rejected immediately. HALF-OPEN: Testing recovery, a few requests pass through.
Failure Detection
The circuit breaker tracks failures over a sliding window. When the failure rate exceeds the threshold within the window, the circuit opens. Common thresholds: 50% error rate over a 10-second window with at least 20 requests.
import time
from enum import Enum
class State(Enum):
CLOSED = "closed"
OPEN = "open"
HALF_OPEN = "half_open"
class CircuitBreaker:
def __init__(self, failure_threshold=0.5, window_size=10,
open_duration=30, half_open_max=5):
self.state = State.CLOSED
self.failure_count = 0
self.success_count = 0
self.total_count = 0
self.window_start = time.time()
self.open_until = 0
self.failure_threshold = failure_threshold
self.window_size = window_size
self.open_duration = open_duration
def call(self, func):
if self.state == State.OPEN:
if time.time() > self.open_until:
self.state = State.HALF_OPEN
self.success_count = 0
self.failure_count = 0
else:
raise CircuitOpenError("Circuit is open")
try:
result = func()
self._on_success()
return result
except Exception as e:
self._on_failure()
raiseThe circuit breaker tracks failures in a sliding window. When the failure rate exceeds the threshold, the circuit opens and rejects all requests for a configured duration.
Envoy Circuit Breaker
circuit_breakers:
thresholds:
- priority: DEFAULT
max_connections: 1024
max_pending_requests: 1024
max_requests: 1024
max_retries: 3
retry_budget:
budget_percent:
value: 20.0
min_retry_concurrency: 3
track_remaining: true
max_ejection_percent: 50Envoy provides circuit breaking at the connection, request, and retry level. The max_ejection_percent prevents the circuit breaker from ejecting all backends.
Hystrix Pattern
Hystrix (Netflix, now in maintenance mode) popularized the circuit breaker pattern. Its concepts live on in Envoy, Resilience4j, and Polly. The key insight: circuit breakers should provide fallback behavior when the circuit is open.
Tuning Circuit Breakers
- Failure threshold — 50% is a good starting point.
- Window size — 10-30 seconds balances sensitivity with stability.
- Open duration — 30-60 seconds gives the dependency time to recover.
- Half-open max — Allow 5-10 requests to test recovery.
- Slow call threshold — Also open on slow calls, not just errors.
The purpose of a circuit breaker is to fail fast. When a dependency is down, stop calling it immediately. This preserves resources, prevents cascading failures, and gives the dependency time to recover.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.