Field notes
Real incidents, real numbers, no listicles.
8posts on SRE, DevOps, backend engineering, and platform engineering — each one a specific scenario with root causes, trade-offs, and the fix that actually worked. Written for engineers who've been paged at 3am, not for search engines.
Graceful Shutdown Isn't Optional: The Rolling Deploy That Dropped 3% of Requests
Every deploy, every day, for months, we silently dropped a small percentage of real requests — and nobody noticed until we actually measured it, because 3% doesn't look like an outage.
- Platform EngineeringJuly 3, 2026
The Day a `kubectl drain` Took Down Payments (and the PDB We Didn't Have)
A routine node drain during a cluster upgrade evicted every replica of a critical service within seconds of each other, because nothing told Kubernetes it wasn't allowed to.
7 min read - BackendJune 29, 2026
The Idempotency Bug That Only Showed Up Under Retries
Our payments API had an idempotency key on every write. It still double-charged a customer, because the bug wasn't in the idea — it was in the four-line race condition between checking the key and writing the result.
8 min read - SREMay 4, 2026
Multi-Window Multi-Burn-Rate Alerts, Explained With the Incident That Made Me Actually Understand Them
Simple threshold alerts either page you for nothing or miss real outages. Here's the incident that forced me to actually understand burn-rate math instead of copy-pasting a Prometheus rule.
9 min read - SREApril 18, 2026
The 3am Page That Taught Me Alerts Need Runbooks, Not Just Thresholds
The alert fired correctly. The threshold was right. I still spent 40 minutes at 3am guessing, because nobody had written down what to actually do about it.
7 min read - DevOpsMarch 22, 2026
Why Our CI Pipeline Took 40 Minutes and What We Actually Cut to Get to 8
No magic caching trick fixed this. It was four separate, boring bottlenecks — and finding them required actually timing every step instead of guessing.
8 min read - DevOpsFebruary 11, 2026
The Rollback That Made It Worse: A Postmortem on Assuming Rollback = Safe
We reverted a bad deploy in 4 minutes, exactly like the runbook said. That's when the real outage started, because the deploy had already changed the database schema and the old code couldn't read the new shape.
8 min read - BackendJanuary 27, 2026
Connection Pool Exhaustion: A Case Study in a 2am Death Spiral
One slow downstream dependency turned into a full outage in six minutes — not because of the slowness itself, but because of what a fixed-size connection pool does when every connection is stuck waiting.
8 min read