The Importer That Changed Its Own Shape Mid-Build

The largest single piece of code on a platform migration I led was a bulk customer importer — built across roughly a week, in six phases, to bring existing organizations and their members into the new system. Partway through, the source of truth for two of those six phases turned out to be wrong, and the importer had to be rewritten to read from somewhere else entirely, without restarting from zero.

What the original plan assumed

The first version assumed organizations and their member rosters could be sourced from one clean, canonical dataset — the natural assumption, because that's what a well-structured export usually looks like. Early phases worked exactly as expected against that assumption: parse the export, create the records, move on.

Where the assumption broke

It turned out the canonical dataset didn't actually contain a reliable picture of which members belonged to which organization — that relationship lived somewhere else, in product purchase records for the organization side and in loosely structured form-response data for the member side, neither of which was designed to be machine-read as a source of truth. Building on the original assumption would have produced organizations with wrong or missing membership, silently, for a meaningful fraction of records.

Rewriting without starting over

The fix wasn't scrapping the importer — most of its phases, the ones dealing with account creation, credential setup, and content, were still correct and still needed. The two phases that were wrong got rebuilt to source organizations from product purchase records instead of the original export, and to derive the member graph from form-response data instead of assuming it lived alongside the organization data. Everything downstream that consumed "here is an organization, here are its members" as an interface kept working unchanged, because the interface between phases hadn't moved — only where two of the six phases got their input had.

The actual takeaway

A large migration script is never really one script — it's a pipeline of phases with an interface between them, and the value of treating it that way shows up exactly when one assumption turns out to be wrong. Rebuilding two phases against a corrected data source, without touching the other four, only works if the phases were separable in the first place. Building for that separability up front is what turns "the data source was wrong" from a full rewrite into a two-phase patch.