Stage 3 · Build

NoSQL & Distributed Databases

Database Migration Patterns

Strangler fig, dual-write, and safely migrating from one database to another.

7 min readDatabases for SREBuild

Migration Approaches

Migrating between databases is one of the riskiest operations in software engineering. The three main approaches are big bang (cutover), strangler fig (gradual replacement), and dual-write (run both simultaneously).

Approach	Risk	Duration	Downtime
Big Bang	High	Days	Minutes to hours
Strangler Fig	Low	Weeks to months	Zero
Dual-Write	Medium	Weeks	Zero

Strangler Fig Pattern

Strangler fig implementation


# Phase 1: Route new features to new database
class UserService:
    def get_user(self, user_id):
         # Old database for existing features
        return postgres_db.query("SELECT * FROM users WHERE id = %s", user_id)

    def get_user_preferences(self, user_id):
         # New database for new feature
        return mongo_db.users.find_one({"_id": user_id, "type": "preferences"})

 # Phase 2: Migrate feature by feature
class UserService:
    def get_user(self, user_id):
         # Check new database first
        user = mongo_db.users.find_one({"_id": user_id})
        if user:
            return user
         # Fallback to old database
        return postgres_db.query("SELECT * FROM users WHERE id = %s", user_id)

 # Phase 3: Remove old database dependency
class UserService:
    def get_user(self, user_id):
        return mongo_db.users.find_one({"_id": user_id})

The strangler fig pattern incrementally replaces database access. Start with read-only features, then migrate writes. Each feature is migrated independently, reducing risk.

Dual-Write Pattern

Dual-write implementation


import threading
import queue

class DualWriter:
    def __init__(self, primary_db, secondary_db):
        self.primary = primary_db
        self.secondary = secondary_db
        self.write_queue = queue.Queue()
        self._start_background_writer()

    def write(self, table, data):
         # Write to primary (source of truth)
        result = self.primary.insert(table, data)

         # Queue secondary write (non-blocking)
        self.write_queue.put(("insert", table, data))

        return result

    def _start_background_writer(self):
        def writer_thread():
            while True:
                operation, table, data = self.write_queue.get()
                try:
                    self.secondary.insert(table, data)
                except Exception as e:
                     # Log and retry failed writes
                    logger.error(f"Secondary write failed: {e}")
                    self.write_queue.put((operation, table, data))
                    time.sleep(1)

        thread = threading.Thread(target=writer_thread, daemon=True)
        thread.start()

Dual-write writes to both databases simultaneously. The primary is the source of truth; the secondary is populated asynchronously. If the secondary write fails, it is retried from the queue.

Dual-write creates consistency windows

Between writing to primary and secondary, data is inconsistent. The application must handle this: read from primary during the consistency window, or accept eventual consistency for secondary reads.

Change Data Capture

CDC with Debezium


# Debezium captures changes from PostgreSQL WAL
 # and streams them to Kafka for consumption by the new database

 # Debezium connector configuration
{
  "name": "postgres-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "primary.example.com",
    "database.port": "5432",
    "database.user": "debezium",
    "database.dbname": "mydb",
    "database.server.name": "mydb",
    "table.include.list": "public.users,public.orders",
    "plugin.name": "pgoutput",
    "slot.name": "debezium_slot"
  }
}

 # Consumer: write changes to MongoDB
class MongoCDCConsumer:
    def process_change(self, change):
        if change["op"] == "c":  # create
            mongo_db.users.insert_one(change["after"])
        elif change["op"] == "u":  # update
            mongo_db.users.update_one(
                {"_id": change["after"]["id"]},
                {"$set": change["after"]}
            )
        elif change["op"] == "d":  # delete
            mongo_db.users.delete_one({"_id": change["before"]["id"]})

CDC captures every INSERT, UPDATE, and DELETE from PostgreSQL's WAL and streams them to the new database. This keeps both databases in sync without modifying application code.

Data Validation

Migration validation script


def validate_migration(source_db, target_db, table, sample_size=10000):
    """Compare source and target databases for consistency"""
     # Get total counts
    source_count = source_db.query(f"SELECT COUNT(*) FROM {table}")
    target_count = target_db.query(f"SELECT COUNT(*) FROM {table}")

    if source_count != target_count:
        print(f"COUNT MISMATCH: source={source_count}, target={target_count}")

     # Sample random rows for comparison
    source_rows = source_db.query(
        f"SELECT * FROM {table} ORDER BY RANDOM() LIMIT {sample_size}"
    )

    mismatches = 0
    for row in source_rows:
        target_row = target_db.query(
            f"SELECT * FROM {table} WHERE id = %s", row["id"]
        )
        if not target_row:
            print(f"MISSING in target: id={row['id']}")
            mismatches += 1
        elif normalize(row) != normalize(target_row):
            print(f"MISMATCH: id={row['id']}")
            mismatches += 1

    print(f"Validation complete: {mismatches} mismatches in {sample_size} rows")
    return mismatches == 0

Always validate after migration. Compare row counts, sample random rows, and verify critical data. Run validation queries on both databases simultaneously to catch in-flight changes.

Rollback Strategy

Keep the old database running during and after migration
Maintain the ability to route reads/writes back to the old database
Test rollback procedure before starting migration
Set a rollback deadline (e.g., 2 weeks post-migration)
After deadline, decommission old database

Feature flags control migration routing

Use feature flags to control which database each request uses. This allows instant rollback by flipping a flag, without code deployment. LaunchDarkly, Unleash, or a simple config file work well.

Migration success criteria

1) Data counts match. 2) Random sample validation passes. 3) Application tests pass against new database. 4) Performance meets or exceeds baseline. 5) Rollback procedure tested and documented.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.