Stage 4 · Provision
Resilience Patterns
Bulkheads and Isolation
Thread pools, connection pools, resource quotas, and cell-based blast-radius reduction.
What Is the Bulkhead Pattern?
The bulkhead pattern isolates components so that a failure in one does not cascade to others. Named after ship bulkheads that prevent a hull breach from flooding the entire ship. In software, isolation means dedicating separate resources (threads, connections, memory) to each dependency.
Thread Pool Isolation
Each dependency gets its own thread pool. If the payment service is slow, it only consumes payment threads — not search threads. This prevents one slow dependency from exhausting all threads.
// Separate thread pools for each dependency
ExecutorService paymentPool = Executors.newFixedThreadPool(20);
ExecutorService searchPool = Executors.newFixedThreadPool(20);
ExecutorService notificationPool = Executors.newFixedThreadPool(10);
// Payment service slow → Only 20 threads affected
Future<Result> paymentResult = paymentPool.submit(() -> callPaymentService());
// Search and notification pools remain unaffected
Future<Result> searchResult = searchPool.submit(() -> callSearchService());Thread pool isolation limits the impact of a slow dependency. The payment service can only consume 20 threads, leaving search and notification unaffected.
Connection Pool Isolation
Database connection pools:
Payment DB:
Pool size: 20
Timeout: 5s
Status: Healthy (5/20 connections used)
Search DB:
Pool size: 30
Timeout: 5s
Status: Healthy (12/30 connections used)
Analytics DB:
Pool size: 10
Timeout: 5s
Status: Exhausted (10/10 connections)
→ Fails fast, does not affect other poolsEach database gets its own connection pool. Exhaustion of one pool does not affect others. This prevents connection starvation cascades.
Resource Quotas
Resource quotas limit CPU, memory, and network per service or tenant. In Kubernetes, resource requests and limits enforce isolation. Without quotas, a noisy neighbor can consume all resources and starve other services.
apiVersion: v1
kind: ResourceQuota
metadata:
name: team-payment-quota
spec:
hard:
requests.cpu: "8"
requests.memory: 16Gi
limits.cpu: "16"
limits.memory: 32Gi
pods: "20"
---
apiVersion: v1
kind: LimitRange
metadata:
name: pod-limits
spec:
limits:
- type: Container
default:
cpu: "2"
memory: 4Gi
defaultRequest:
cpu: "1"
memory: 2GiResource quotas prevent one team from consuming all cluster resources. LimitRanges set default resource requests and limits for containers.
Cell-Based Architecture
Cell-based architecture divides your system into independent cells, each serving a subset of users. A failure in one cell does not affect others. This is the most powerful blast-radius reduction technique. AWS and Slack use cell-based architectures.
User Request
│
▼
Cell Router (maps user → cell)
│
├── Cell A (users 1-1M)
│ ├── App instances
│ ├── Database
│ └── Cache
│
├── Cell B (users 1M-2M)
│ ├── App instances
│ ├── Database
│ └── Cache
│
└── Cell C (users 2M-3M)
├── App instances
├── Database
└── Cache
Failure in Cell B → Only users 1M-2M affected
Other cells continue operating normallyEach cell is a complete, independent copy of the service. Cells are isolated at the data, compute, and network level. A cell failure affects only its assigned users.
Blast Radius Reduction
- Cell-based — Isolate users into independent cells.
- Bulkhead — Isolate resources per dependency.
- Circuit breaker — Stop calling failing dependencies.
- Timeout — Prevent slow calls from consuming resources.
- Load shedding — Drop low-priority work under pressure.
Bulkheads provide 80% of blast-radius reduction with 20% of the complexity. Implement thread pool and connection pool isolation first. Graduate to cell-based architecture when you need stronger isolation guarantees.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.