What Would Make This Fail?
Last week I ran a document through a linter I wrote myself. When it has no complaints, it prints one word: CLEAN. That’s what it printed. Then I gave the same document to a second reviewer, a model that reads the document rather than scanning it for patterns. It found an arithmetic error a few thousand dollars off.
The linter was working correctly. It checks prose style, and arithmetic was never inside it. It reported precisely what it was built to report.
CLEAN meant the things I check are fine. I read it as the document is fine. A check tells you one dimension passed. Everybody reads it as the work being fine. The gap between those two claims is where my error was living.
I only caught it because I happened to send the document on to something else. Left to myself, CLEAN is where I stop reading.
None of this started with AI. In January of 2007 I wrote a post about Earned Value Management and complained that EVM has no method of measuring project quality, only progress. A project can be inside scope, under budget and on schedule, and the customer can still consider the whole thing a failure, and there’s nothing in EVM that will tell the project team so. The measurement is honest about what it measures and silent about everything else. The person reading the report supplies the missing confidence out of their own pocket. What has changed since 2007 is how many of these things are running at once, and how rarely anyone reads what any of them actually covers. (Back in July I wrote about the same trouble from the other side, when polish got cheap and looking finished stopped meaning checked.)
What a green check does to attention
A gate that reports green removes somebody’s reason to look, and that’s most of the point of installing it. It doesn’t have to replace anyone to do this. The reviewer downstream is still employed and still capable, sitting exactly where they were. They see the pass, believe the area is handled, and spend their attention somewhere else, which is the behavior you were buying.
So coverage goes up and attention goes down at the same time. Coverage shows up on a dashboard. There’s no dashboard for the attention, so nobody notices it leaving.
Sometimes none of that matters. A checksum, an invoice matched against a purchase order, a filing that won’t submit until every field is populated: the scope of the check is the whole of the work, so a pass really is a verdict and reading it as one is correct. The trouble is the check whose scope is narrower than the work it sits in front of, and whose result says nothing about the difference. Mine printed one word. The list of things it never looks at exists, but it lives in the source code, not in the green light.
That’s what I was missing, and asking for it is what changed in how I work. A pass is a statement about a specific set of conditions rather than about the document, so the first thing I do with a green result now is ask what sat outside those conditions. A second linter would have returned CLEAN on that document too, and so would a third. Stacking more checks of the same kind buys nothing except a stronger feeling. What caught the arithmetic was a reviewer built to do something else entirely.
Most people reading this don’t run a linter. They’ve approved something else in the last eighteen months: AI review running on every pull request, automated contract review, a compliance gate that clears vendors, a dashboard that turns green on a schedule and gets glanced at in a Monday meeting. In every one of those cases somebody said that area is covered now, and the word covered is doing the identical work that the word CLEAN was doing in my terminal.
Vendors always answer “what does this check look at” on the marketing page, and the answer is always impressive. Ask instead what input would turn it red. Some tools can tell you precisely. Many of the AI review products sold in the last two years cannot produce the list at all, because there isn’t one to produce, and a gate whose failure conditions nobody can state is a gate whose coverage nobody can bound.
What the answer will not give you is a place to look. Invert my linter’s fail list and you don’t arrive at “check the arithmetic.” You arrive at the whole rest of the world: the facts, the numbers, the law, whether any of the argument is true. The list tells you how small a claim the green light was making. That is the entire value of asking, and it’s worth the ten minutes.
Which leaves a duller answer than I wanted about what to actually do. What caught the arithmetic was not me looking harder in the right place, because I had no way to know where that was. It was a reviewer built to do something else entirely, reading the same document. More checks of the same kind buy a stronger feeling. One of a different kind buys coverage, and that is a purchasing decision rather than a discipline problem.
I don’t have a clean rule for how many differently-shaped checks are enough, and I doubt one exists. What I have is a habit: when something comes back green I ask what it was built to see, and then I go find out who is looking at everything else.