Stage 4 · Provision
Cache Design Patterns
Thundering Herd Control
Request coalescing, probabilistic early refresh, locks, and stale-if-error responses.
What Is the Thundering Herd Problem?
The thundering herd problem occurs when a large number of requests simultaneously hit a resource after a cache miss or event. If 10,000 users request the same uncached resource simultaneously, all 10,000 requests hit the database. This can crash the database or cause cascading failures.
Request Coalescing
Request coalescing collapses multiple identical concurrent requests into a single request. When the first request starts loading data from the database, subsequent requests for the same key wait for the first request to complete rather than each independently querying the database.
import redis
import time
import json
r = redis.Redis()
def get_with_coalescing(key: str, loader, ttl: int = 300):
# Check cache first
cached = r.get(key)
if cached:
return json.loads(cached)
# Try to acquire lock (only one request loads)
lock_key = f"lock:{key}"
acquired = r.set(lock_key, "1", nx=True, ex=10)
if acquired:
try:
# This request is the loader
data = loader()
r.setex(key, ttl, json.dumps(data))
return data
finally:
r.delete(lock_key)
else:
# Another request is loading — wait and retry
for _ in range(50): # Wait up to 5 seconds
time.sleep(0.1)
cached = r.get(key)
if cached:
return json.loads(cached)
raise TimeoutError("Cache load timeout")Only one request loads data from the database. All other requests wait for the result. This reduces database load from 10,000 queries to 1 query.
Probabilistic Early Refresh
Probabilistic early refresh randomly refreshes hot cache entries before they expire. This prevents all requests from hitting the database simultaneously when the TTL expires. A small percentage of requests trigger a refresh, warming the cache for subsequent requests.
import random
def get_with_early_refresh(key: str, loader, ttl: int = 300):
cached = r.get(key)
if cached:
# Probabilistic refresh: 1% chance to refresh before expiry
remaining_ttl = r.ttl(key)
if remaining_ttl < ttl * 0.1: # Less than 10% TTL remaining
if random.random() < 0.01: # 1% probability
data = loader()
r.setex(key, ttl, json.dumps(data))
return data
if cached:
return json.loads(cached)
# Cache miss — load data
data = loader()
r.setex(key, ttl, json.dumps(data))
return dataWhen the cache entry is near expiry, 1% of requests trigger a refresh. This keeps the cache warm without all requests hitting the database simultaneously.
Distributed Locks
Distributed locks ensure that only one instance loads a cache miss. Redis SET NX provides atomic lock acquisition. The lock has a timeout to prevent deadlocks if the holder crashes.
Stale-If-Error
Cache-Control: max-age=60, stale-if-error=86400
Behavior:
Normal: Serve fresh content (within max-age)
Error fetching fresh content: Serve stale content for up to 24 hours
After 24 hours: Return error instead of stale contentstale-if-error tells the CDN to serve stale content when the origin is unavailable. This provides graceful degradation during origin outages.
Prevention Strategies
- Request coalescing — Collapse concurrent requests into one.
- Probabilistic refresh — Randomly refresh before expiry.
- Cache warming — Pre-populate cache before traffic spikes.
- stale-if-error — Serve stale content during outages.
- Jittered TTLs — Add random jitter to prevent synchronized expiry.
If all cache entries have a 300s TTL, they all expire at the same time. Add random jitter: TTL = 300 + random(0, 60). This spreads cache misses across a 60-second window instead of a single moment.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.