A 51M-record migration with zero downtime

The client: a healthcare SaaS platform migrating customer data onto a new platform architecture.

The problem

A large volume of customer data — north of 51 million records — needed to move onto a new platform architecture. The previous process was manual and took weeks per customer. It also had to happen without downtime: this was live operational data for businesses that don’t stop running during a migration.

Approach

The migration ran as a batched, multi-threaded workload on Kubernetes, moving data in controlled chunks rather than one long-running job — so a failure mid-migration meant retrying a batch, not starting over. Idempotency was the load-bearing property throughout: every batch could be safely re-run without corrupting state, which is what made near-zero-touch automation possible instead of someone babysitting a script.

The pipeline ran repeatedly in production as it scaled across the customer base, with each run validated before the next batch proceeded.

Outcome

What used to take weeks of manual work per customer now runs as an automated pipeline, with zero downtime for the businesses whose data was moving.

What I’d tell you if you were building this

For a migration at this scale, idempotency isn’t a nice-to-have — it’s what turns “the job failed at 2am” from an incident into a non-event. Design for safe re-runs before you design for speed.