Resumable Batch Migrations

Moving data between two systems, whether CMS platforms, databases or SaaS tools, is a long-running job on unreliable networks. These are the patterns that make it survivable. I learned most of them building a CMS migration tool (A Resumable CMS Migration Tool).

1. Extract first, load later

Export everything from the source into a staging area (local disk, object storage) before writing anything to the target. This decouples the two systems: the source can be left alone, the export can be inspected, and the slow, risky load can be retried without touching the source again.

2. Turn the work into a queue

Scan the staging area and build an explicit queue of jobs, ordered by dependency (types before entries, assets before the entries that reference them). A queue makes progress visible and failure local: one bad item fails one job, not the run.

3. Stream in batches; don’t load everything

Loading a whole dataset into memory works on small inputs and falls over on real ones. Process the queue in batches and stream: read a slice, transform, write, release, repeat. Memory stays flat regardless of size, and batch size becomes a tuning knob for rate limits. It’s the same backpressure idea that keeps a busy UI smooth (Message Passing, Zustand Transient Updates).

4. Adapters between models

Source and target never share a model. Put an adapter (an anti-corruption layer) between them that maps one interface onto the other: field types, rich text, references, enums, odd inputs. The engine stays generic and each new source is just a new adapter. This is the classic Adapter pattern earning its keep.

5. Checkpoint and make writes idempotent

Long runs fail. Checkpoint each completed item so a restart resumes where it stopped. Make every write idempotent by keying on a stable source ID, so replaying a batch updates rather than duplicates. Together these turn “it crashed after three hours” from a disaster into a shrug.

6. Dry runs and reports

A dry-run mode that reports what would be created, changed or skipped, and an end-of-run report of failures with reasons, make migrations reviewable by people who don’t read code.

Where it runs

On a laptop, resumability is essential because the machine sleeps, the Wi-Fi drops and memory is tight. In the cloud, the same design maps neatly onto queue-triggered functions or workers that scale out and run around the clock. And today, the most manual step, mapping one model onto another, is a good job for an AI assistant to draft and a human to review.

Related: A Resumable CMS Migration Tool · Message Passing · Types of Failure · Node.js Design Patterns