How to Tell What's Real in AI
Lately a lot of my time is spent in two rooms that don’t describe AI the same way. In one, I’m building the thing: training models, running evaluations, digging through a production system when the verdicts come back wrong. In the other, I’m across the table from people who have to make the call on AI they can’t build themselves, what to buy, what to trust, what to bet real money on. It took me a while to notice how little those two rooms overlap when they’re talking about the very same technology.
In the ‘building room’, nobody talks the way a vendor deck talks. I’ve never once heard an engineer say “this model is 99% accurate” and just leave it there. Somebody always pushes: measured on what data? At what threshold? What happened on the cases it got wrong? That confident single number that anchors a sales meeting wouldn’t last ten minutes in a model review. The people closest to the technology are the ones who trust the bold claims the least.
In the ‘buying room’, the people signing the checks, get the deck. One number, one demo, one case study with a suspiciously clean improvement number. And most of the advice written for them is either a giant procurement checklist (data residency, SOC 2, encryption at rest, all fine, all necessary, and none of it the thing that decides whether the AI actually works) or one more take on whether AI is over-hyped or under-hyped, which helps nobody decide anything.
So I’ve been trying to write down what actually closes that gap, because I’m pretty sure it’s learnable and almost none of it needs code. It comes down to five questions I’ve been asking for years. Here they are.
One: What exactly was measured, and by whom?
Every number a vendor shows you is an answer to a question, and they picked the question. “Resolves 70% of tickets” hangs entirely on what counts as resolved and what got left out of the denominator. You need the definition under the number before the number means anything, and most impressive figures get a lot less impressive once you find it.
Two: What happens on our data?
Benchmarks run on clean, curated inputs. Yours aren’t. In my own work, the distance between how a model does on its test data and how it does on the messy stuff it actually meets in production has been the single most reliable source of disappointment. So nothing a vendor shows you is real until some version of it has run on a sample of your own cases, the ugly ones included. A vendor who’s confident will have a way to let you do that. And if they won’t, I’ve learned to treat that as an answer in itself.
Three: Where does it break?
Every system has a failure map, the specific inputs where it falls apart. Engineers obsess over this and sales decks never bring it up, which should tell you something about how useful it is. An average score tells you how often it fails. The failure map tells you where, and that’s the information that you need to actually price the risk.
Four: What is it standing on?
A model is only as good as the data and plumbing under it, and I’ve watched a production system’s problems get pinned on the model for weeks when the model was fine and the floor under it wasn’t. Ask what the system assumes about your inputs, and what happens when those assumptions don’t hold. When one of these things underperforms, the model is the obvious suspect and rarely the guilty one, which is exactly why you judge the whole stack and not just the part with “AI” in the name.
Five: Which mistakes can we live with?
This is the one that turns the rest into a decision. Every AI system makes errors, and what you’re really choosing is which ones, how often, in exchange for what. A tool screening video interviews and a tool approving wire transfers shouldn’t tolerate the same mistakes even if the model underneath is identical. That call is a business judgment, and it’s yours, not the vendor’s.
Asking the five questions doesn’t take a line of code. What it takes is harder to come by: the patience to keep asking what a big number actually means before you let it make the decision for you, which is genuinely uncomfortable in a room where everyone else sounds sure.
Honestly, part of me still wants to believe the number and move on, even after years of this. I just know where that road ends, which is buying the deck and finding out the hard way, months later, that it was never measured for you.