Srikanth Bandaru

A migration is done when the last reader is gone

A week of quiet query logs won’t show you the quarterly job, the open tab or the warehouse copy that still reads the column you’re about to drop.

Contents4 sections
  1. Watching proves only what you watched long enough
  2. The readers you forget you have
  3. Each reader needs a proof
  4. Fill in a reader ledger

Every guide to changing a database schema without downtime ends with the same step. Remove the old column once all clients have moved. The guides don’t say how you’d know. You’re only done when you can prove, one reader at a time, that nothing still reads it.

The pattern is called expand/contract, and it has 3 steps. Step 1 adds the new column beside the old one (expand). Step 2 moves every reader and writer over. Step 3 removes the old column (contract). Danilo Sato’s ParallelChange puts step 3 like this. “Once all usages have been migrated to the new version, you perform the contract phase.”

The guides treat “once” as a date you look up. It’s a claim about every reader. So for an old column called legacy_email, the useful question is who reads it, and what proves each one is gone.

Watching proves only what you watched long enough

The tempting proof is telemetry. Watch for reads of the old column, see none, drop it. Apollo’s GraphOS schema checks do this for GraphQL fields, over 7 days of traffic by default. That’s fine for an hourly reader. Pick a window, drop the column, and see what a quarterly job does.

How long did you watch?

Model: readers and schedules invented for this figure

Pick how many days to watch the query log for reads of legacy_email. Then drop the column.

Each row is a job that reads legacy_email on a schedule. Shaded: the days you watched. A job you didn’t see breaks at its next run. The schedules are invented; the rule P(seen) = min(1, window / period) is not.

A 1-week window catches a quarterly job about 8% of the time. A query log only sees readers that query the database while you watch. Many readers hold their own copy of the column, or of code that expects it.

The readers you forget you have

Your own servers. During a rolling deploy, servers that booted before the drop keep the old column list. Active Record, the Rails library that maps tables to objects, caches columns at boot. That’s why strong_migrations makes you ignore a column for one deploy before you drop it. Prepared statements do the same without any library. Postgres fails a SELECT * prepared before the drop with cached plan must not change result type. That query never named the column.

Open browser tabs. A tab that loaded yesterday’s code may still be running it. Malte Ubl’s note on version skew says tabs stay open “for days or even weeks.”

The warehouse. Change data capture (CDC) copies each row change out of the database’s write-ahead log, its record of every change. Schema changes like ALTER TABLE don’t reach that stream. Debezium’s Postgres connector says it can’t report them (“DDL change events”) to consumers. Fivetran leaves a deleted column “untouched in your destination” and fills it with NULL. Every dbt model (a SQL transform in the warehouse) and dashboard on that table keeps reading it, and the database can’t see them.

Each reader needs a proof

Clear each reader with one of 4 proofs, or decline to clear it.

Proof What it is How it fails
Wait The reader has a known maximum lifetime, so you outwait it (pin max age, topic retention, rollback window) The lifetime was a guess, or there wasn’t one
Watch Telemetry shows no reads for long enough The window was shorter than the rarest reader’s period
Evict Force the reader off the old column (reload on version mismatch, mobile force-update, brownouts, hard errors on old reads, Deprecation and Sunset headers) Someone has to own the eviction and what it breaks
Keep Keep serving the old column on purpose (Stripe’s API versions, dbt model versions) New code starts to depend on the old column

Evict is the only proof for readers with no known lifetime. GitHub ran brownouts, short planned outages of an old endpoint, before removing it. Stripe’s migration write-up puts a hard error on the old read path, so someone sees every leftover read.

Keep is fine if you choose it on purpose. A kept column needs its writes kept too, or old readers get stale values with no error.

Fill in a reader ledger

Before the contract step, I want one row per reader, with its proof. Here is an example.

Reader Proof What the proof rests on
API servers, workers Wait The rollout ends and every process restarts
Open tabs Evict Reload when the tab’s version doesn’t match the server’s
Mobile apps Evict A forced update, so no old build still runs
Kafka topics (message streams) Wait Retention removes the old messages
dbt models, dashboards, exports Watch Query history for at least the rarest job’s period
CDC copy in the warehouse Evict Drop the warehouse column once the row above is clear
External API clients Keep Serve the old field to clients on old versions

Also stop the list from growing while you work. When I moved the on-screen text of sales apps in 5 markets into a translation service, I added a CI check. It fails on hardcoded copy, meaning text typed straight into the code. A second change made the service the only source and removed about 20,000 lines of local scripts.

For a column, the same check is a CI rule that fails on any new reference to legacy_email.

If a row has no proof, the migration isn’t done. It’s still at step 2, whatever the plan says about step 3.