DevOps
The Rollback That Made It Worse: A Postmortem on Assuming Rollback = Safe
We reverted a bad deploy in 4 minutes, exactly like the runbook said. That's when the real outage started, because the deploy had already changed the database schema and the old code couldn't read the new shape.
"Just roll back" is the most common instinct during an incident, and it's usually correct. This is the story of the one time it wasn't — where rolling back a bad deploy turned a minor, contained bug into a full read/write outage, because we'd forgotten that a deploy isn't just code changing. Sometimes it's the database changing too, and code can roll back a lot faster than a schema can.
The deploy that looked routine
A normal feature release: add a status column to the orders table to support a new order-tracking feature, and ship application code that reads and writes it. The migration ran first (add column, backfill default value, done in under a minute), then the new application code deployed. Five minutes after the new code was live, error rates on the orders service ticked up — not catastrophically, maybe 3% of requests, all related to a null-handling edge case in the new tracking logic. Textbook "revert the bad deploy" situation.
The rollback: fast, correct-looking, and wrong
We did exactly what the runbook said: redeploy the previous known-good application version. It took 4 minutes end to end — genuinely fast, genuinely by-the-book. And the moment it finished, error rates didn't drop back to baseline. They spiked to nearly 30%, on a completely different set of endpoints than the original bug.
The old application code had no idea the new status column existed, but that wasn't the immediate problem — extra columns are usually harmless to old code. The real problem was subtler: the new code's migration had also changed a NOT NULL constraint on a related column as part of the same release, and the old code had never been written to satisfy that constraint. Rolling back the app code without rolling back the schema meant old code was now writing rows that violated a constraint only the new code knew how to satisfy.
Why "rollback = safe" is a dangerous default assumption
Application code rollback is close to instantaneous and (usually) safely reversible — old binaries, old configs, no lasting state change. A database migration is not the same kind of operation. Once it's run, the schema is the schema for every connection, including ones from application versions that predate the migration. Rolling back code doesn't roll back the database — you'd need a *down* migration for that, and most teams (we included, at the time) don't reliably write or test those.
| Rollback type | Actual reversal time | Safe by default? |
|---|---|---|
| Application code (stateless) | Seconds to minutes | Usually yes |
| Feature flag | Seconds | Usually yes |
| Additive schema change (new nullable column) | N/A — old code ignores it | Usually yes |
| Constraint change / column drop / type change | Requires a tested down-migration | No — assume unsafe until proven otherwise |
What actually fixed it (and it wasn't a rollback)
Once we understood the real cause, the fix was to roll *forward*, not back: patch the specific null-handling bug in the new code (a small, well-understood change) and redeploy the new version, leaving the schema untouched. That resolved the incident in about 12 minutes from diagnosis to green — faster than the original "rollback," and without touching the database at all.
The fix for this class of problem isn't discipline about remembering — it's structuring migrations so this situation is never possible. Expand/contract means: (1) add the new column as nullable, deploy code that can read both old and new shapes, (2) backfill, (3) only once the old code path is fully retired, add the NOT NULL constraint or drop the old column in a separate, later release. Every step in isolation is safely rollback-compatible with the application version that preceded it.
-- Release 1: purely additive, safe to roll back at the app layer
ALTER TABLE orders ADD COLUMN status TEXT; -- nullable, no default constraint yet
-- deploy app code that writes status but tolerates it being null on read
-- (wait for release 1 to be fully stable, at least one full deploy cycle)
-- Release 2: only now, once nothing depends on status being nullable
ALTER TABLE orders ALTER COLUMN status SET NOT NULL;
ALTER TABLE orders ALTER COLUMN status SET DEFAULT 'pending';If release 2 needs to be rolled back, the app just reverts to code that tolerated a nullable status column — which the schema still technically allows until release 2's constraint is applied. There's no version of the app that's incompatible with the schema at any single point in time.
The one question we now ask before every release
"If we have to roll back the application code in the next 10 minutes, is the current database schema still fully compatible with the previous app version?" If the honest answer is no, the migration needs to be split — not because rollback might be needed, but because if it is needed, you want it to actually work.
- Constraint changes (NOT NULL, foreign keys, unique) go in a separate, later release from the code that requires them.
- Column drops happen only after confirming zero running app versions still read that column.
- Every migration gets a written, tested down-migration — even if you don't expect to use it, because it forces you to think through exactly this failure mode.
- "Rollback" during an incident means: check if the previous release touched schema first. If it did, don't assume revert-the-code is the safe move — it might be the one that makes it worse.