The Verification Problem

A few weeks ago, I had AI rebuild a chunk of one of my products. Two coding agents running at once with full test coverage. Every test passed. Then I sat down and went through it by hand anyway, and found that a new customer without a subscription couldn’t actually sign up. The one thing that makes the business money was broken, and nothing in the automated suite said a word about it.

For your entire career, a well-designed and finished-looking piece of work told you something. A clean presentation deck, a tight analysis, no typos, a memo that read like the person knew what they were doing. You couldn’t personally re-check every number, so you let how finished it looked stand in for how carefully it was done. That mostly worked, because making something look that buttoned-up took real effort and real time. The finished look was evidence of the work behind it. AI has made that look free, so ’looking finished’ no longer serves as evidence of anything.

Now anyone on your team can produce something that looks finished in a few minutes, whether they checked it or not, whether they understood it or not. Researchers at Stanford and BetterUp gave this a name last year. They call it workslop : work that looks polished but has no real substance underneath. In their survey, about four in ten employees said they’d received some in the past month, and many trusted the sender a little less afterward. The signal you’d normally use to catch sloppy work, the way it looks, is exactly what AI defeats.

What that does to a company is more dangerous than people think. The way your organization checks work was built for a world where output was expensive. Things moved at the speed a person could produce them, slowly enough that a second set of eyes usually spent some time on the work somewhere along the way. You didn’t design most of that; it was just there as a side effect of work being slow.

AI took out the friction, and the checking that used to ride along for free went with it. The output got faster while the verification stayed exactly where it was (and perhaps even got slower).

I think that’s a big part of why so many AI efforts quietly underperform. MIT’s research group looked at 300 enterprise AI deployments last year and found that 95% of them delivered no measurable impact on the bottom line . The usual read is that the models aren’t good enough, which I don’t buy. The models are genuinely capable; what’s missing is the part nobody has been able to bottle up and sell a license for: the judgment about what to trust and where to look.

Back to my signup bug. I didn’t catch it by being clever. I had two agents do the work, one writing the fixes and one reviewing them, which is itself a kind of cheap, automated checking, and it’s a good thing to do. I read the code myself, had an agent write a full test plan with unit tests, integration tests, and end-to-end browser tests. All tests passed. And I still sat down and used the thing by hand, clicking through it like a customer would, because the product takes people’s money and I’ve learned not to trust a green checkmark. That manual pass is where I found it.

A passing test confirms the software does what I told it to do but it says nothing about whether what I told it to do was right.

Software is the one place in your company with mature, automated verification. Decades of tooling exist for nothing but checking software, and it still wasn’t enough on its own. That said, your marketing has no test suite and neither does your financial model. The market analysis that’s about to shape a real decision has no checkmark that turns red when it’s wrong. If the most heavily instrumented function in the building still needed a human deciding where to look, the functions running on nothing but a finished look need it far more.

This turns verification into something you have to design on purpose. It means deciding, function by function, where a human stays in the loop, how deep the checking goes, and who’s accountable for the finding, sorted by what’s actually at stake. A throwaway internal summary and the analysis behind a real decision don’t deserve the same scrutiny, but AI makes them arrive looking equally finished, so the same quick glance tends to land on both.

Deciding that, on purpose, is the work, and it stays your work. You can’t buy it as a product, and you certainly can’t hand it back to the tool that produced the output in the first place.

What makes this hard to manage is that none of it announces itself. When verification disappears, you don’t get an error; you get a clean document that looks like every other clean document.

The warning I got was a luxury but most of the time there’s no test to fail, just a quiet gap between what looks done and what is, and the only thing standing in that gap is whether someone decided to look.