Zero-Downtime Migrations

Why Database Migrations Are Hard

Database migrations are one of the riskiest operations in production. Unlike application code, which can be rolled back instantly, database changes involve schema modifications, data transformations, and coordinated rollouts across multiple services. A failed migration can cause data loss.

Dual-Write Pattern

Dual-write writes to both the old and new databases simultaneously. During migration, all writes go to both stores. Once the new store is verified, reads are switched over. This provides a safe migration path with the ability to fall back to the old store.

class DualWriter:
    def __init__(self, old_db, new_db):
        self.old_db = old_db
        self.new_db = new_db
    
    def write(self, key, value):
        # Write to both databases
        self.old_db.put(key, value)
        
        try:
            self.new_db.put(key, value)
        except Exception as e:
            # Log but don't fail — old DB is source of truth
            log.error(f"New DB write failed: {e}")
    
    def read(self, key):
        # Read from old DB (source of truth during migration)
        return self.old_db.get(key)

CDC-Based Migration

CDC-based migration uses Change Data Capture to stream data from the old database to the new one in real-time. This is more reliable than dual-write because it captures all changes atomically from the old database's transaction log.

Phase 1: Initial sync
  Old DB ──── full dump ────► New DB

Phase 2: Stream changes
  Old DB ──── CDC (Debezium) ────► Kafka ────► New DB consumer

Phase 3: Verify consistency
  Compare row counts, checksums between old and new

Phase 4: Switch reads
  Application reads from new DB

Phase 5: Stop old DB writes
  Decommission old DB

Strangler Fig Pattern

The strangler fig pattern gradually replaces parts of an old system with new components. Instead of migrating the entire database at once, you migrate one service at a time. Each service migrates its own data to its own database,逐步 strangling the old monolithic database.

1. Identify a service to migrate (start with the least critical).
2. Create a new database for that service.
3. Dual-write to both old and new databases.
4. Switch the service to read from its new database.
5. Stop writing to the old database for that service's data.
6. Repeat for the next service.

Schema Migrations

Schema migrations must be backward-compatible. The old application code must work with the new schema, and the new application code must work with the old schema. This is because deployments are gradual — not all instances update simultaneously.

-- Phase 1: Add column (nullable, no default)
ALTER TABLE users ADD COLUMN phone VARCHAR(20);

-- Phase 2: Deploy code that writes to new column
-- Phase 3: Backfill existing rows
UPDATE users SET phone = '' WHERE phone IS NULL;

-- Phase 4: Add NOT NULL constraint
ALTER TABLE users ALTER COLUMN phone SET NOT NULL;

Rollback Strategies

• Forward-only migrations — Only go forward, never rollback. Fix forward.
• Blue-green databases — Keep old database running until migration is verified.
• Backup before migration — Always take a backup before any schema change.
• Feature flags — Gate new database usage behind flags for instant rollback.