It passed CI, then silently failed to apply in production

There's a specific kind of production incident that doesn't announce itself. No alert fires. No on-call page. No stack trace to grep through at 2am. The schema just quietly isn't what anyone thinks it is, and you won't find out until something downstream starts behaving strangely in a way that takes three people and a long afternoon to trace back to its source.

This is that incident. Or more precisely, this is the setup for it.

The Setup Looks Fine. That's the Problem.

Two feature branches. Two migrations. Both reviewed, both merged, CI green on both.

The problem is sequencing.

Branch A adds a column. Branch B adds a constraint that depends on that column existing. Branch B merged first because the PR was small and got approved quickly. Branch A merged second, even though it was written first.

The deploy pipeline took the order without complaint. It ran Branch B's migration against a schema where the column didn't exist yet. Depending on the database and how the migration was written, it either no-op'd silently, or threw a soft error the pipeline swallowed and kept moving. Then Branch A ran. The column got added. Everything looked fine.

Except the constraint was never applied.

Nobody noticed. CI had been green the whole time. Nobody was going to notice until something relied on that constraint and discovered, at the worst possible moment, that it wasn't there.

What CI Actually Validated

Here's the part that stings during a postmortem.

CI validated that each migration was well-formed in isolation. It checked syntax. It ran each file against a clean schema and confirmed it didn't blow up. That is a real check. It's useful. Nobody is disputing that.

What it did not do is validate that both migrations, applied in the order they actually landed in production, produced the intended end state. That's a different check. Most pipelines don't run it.

The gap sounds theoretical until you're staring at a production schema that's missing a constraint that three downstream services are implicitly relying on. Then it stops sounding theoretical and starts sounding like the opening sentence of a very uncomfortable incident review. It's like building a bridge one bolt at a time, inspecting each bolt individually, and only discovering the bolts needed each other when the whole thing sags in the middle.

Why Concurrent Branches Make This Worse

When one person owns all the migrations, sequencing usually works out. They know what order things need to happen. They write them that way.

Concurrent branches break that immediately.

Two engineers, two branches, neither one fully aware of what the other is doing at the exact moment they're both writing migrations. The migrations can be logically dependent even if the engineers don't realize it. Or one of them does realize it, rebases to account for it, and then the other branch merges first anyway and the rebase is suddenly irrelevant. You could say the rebase went from a safety net to a shrug — a lot of careful work undone by timing.

The process has no mechanism to catch this. Review happens per-PR. CI runs per-commit. Neither step has any visibility into how multiple migrations from multiple branches will interleave in the final merge order.

That's the gap. It's not a gap in individual diligence. It's a gap in what the process is even capable of seeing.

What an Explicit Ordering Check Looks Like

There are a few ways to close this gap. None of them are complicated. They just have to actually exist.

Rebase-before-merge requirements. If a PR touches a migration file, it has to rebase against the latest migration state on main before it can merge. This forces the author to see what's landed since they branched and verify their migration still applies correctly on top of it. It doesn't prevent bad sequencing on its own, but it makes the sequencing visible. That's a meaningful improvement over the current default.

Migration sequence linting. A linter that checks, at merge time, whether the full ordered sequence of pending migrations produces a valid schema. Not "does this file parse" but "does the whole stack, in order, end up where we expect." More work to build. Actually catches the class of bug we're talking about.

Integration tests against cumulative schema state. Instead of running migrations against a clean baseline, run them against a snapshot of the current production schema, in the exact order the deploy pipeline will use. If the result fails to match the expected end state, the test fails. This is the most thorough option. It's also the most expensive to maintain, so it's worth being honest about the tradeoff before committing to it.

Any one of these is better than the current default, which is "CI passed, so the schema must be correct."

The Diligence Point

When you're auditing a codebase, migration history isn't just documentation. It's a record of sequential state changes that have to be evaluated as a sequence, not as a collection of individually valid files.

If the process that produces that sequence can be influenced by merge order, PR review timing, or branch interleaving, and there's no check that validates the sequence as a whole, then "CI passed" is not evidence of schema correctness. It's evidence that each file was syntactically valid. Those are not the same thing, and conflating them is how the constraint goes missing.

The question to ask in a technical audit isn't whether migrations were reviewed. It's whether the system is capable of detecting a dependency violation between two migrations from concurrent branches before it reaches production.

If the answer is no, the process is relying on luck and coordination. In a codebase with multiple contributors and active feature development, that's not a process. That's a hope.