Why "It Passed Staging" Isn't a Migration Signal

# Why "It Passed Staging" Isn't a Migration Signal A database migration ran clean in staging three times before the team scheduled the production cutover. It failed on the first attempt in production, on a table staging didn't have a large enough copy of to expose the problem. The migration wasn't wrong. Staging was answering a question the team hadn't actually asked. ## Staging environments are a proxy, and the proxy has a specific gap The staging database for this system was seeded from a scrubbed, sampled subset of production data — a normal, sensible setup for a staging environment that also needs to respect data privacy. The sampling meant staging's largest table was roughly four percent the size of its production counterpart. The migration in question added an index with a lock-acquisition step that, at staging's row count, completed in under a second. At production's actual row count, the same operation held a lock long enough to start queuing writes, which cascaded into request timeouts across an unrelated service that shared the same database. "It passed staging" was a true statement about a real test that had actually run. It just wasn't a statement about the thing that mattered, because the property that determined success — how long a lock gets held — scales with data volume, and staging's data volume was never representative of production's by design, not by oversight. ## The gap isn't always size — sometimes it's shape Size is the most common version of this gap but not the only one. A different migration in the same environment passed staging cleanly and then hit a constraint violation in production, because a data quality issue that had accumulated over years of production usage — a handful of rows with a null in a column the new constraint required to be non-null — simply didn't exist in the sampled staging data. Staging wasn't just smaller, it was cleaner, in a way that happened to make it blind to exactly the row shapes a real migration needs to survive. ## What actually derisks a migration Passing staging is necessary and not close to sufficient. What closes the gap is running the migration against something that matches production on the specific dimension the migration is sensitive to — which requires first identifying what that dimension is, rather than assuming "staging passed" generalizes. For a lock-duration risk, that means testing against a production-scale row count, even if that has to be a disposable full-size clone rather than the everyday staging environment. For a data-quality risk, that means running a dry-run query against actual production data — read-only, no write — to check for the specific violating rows before the migration that would fail on them ever runs for real. Neither of those checks is exotic. Both are more expensive than trusting a staging pass, which is exactly why staging passes get trusted as a substitute more often than they should. The honest framing for a migration review isn't "did it pass staging" — it's "what does staging fail to represent about production, and did we test that specific gap directly." Every staging environment has at least one such gap, usually more than one, and the ones that cause real incidents are never the ones the team already knew to worry about. They're the ones nobody thought to check because the proxy had, until then, always been good enough.