The Score Problem
A while back the project board for one of our tools listed a ticket as a Priority 1 (P1). It was one P1 among nine. Somewhere else, in the running notes the team kept, a developer had written three days earlier that he was blocked on this ticket and couldn’t use the tool on our own work until it moved. The board knew the ticket mattered, and the notes knew a person was stopped behind it. Nothing joined those two facts except someone happening to read both.
That P1 was everything the board knew about the ticket. In a status review, we’d have looked at it and moved on to the next one.
I’ve been thinking about that ticket since I ran across a post by Amrut Joshi in October called Asking LLM for confidence. He’s writing about a worse version of the same thing, where the number may not even be right. You ask a model to do a task, then ask how confident it is, and it hands back a number. His read on that number: “‘0.85’ is a plausible continuation of the conversation. It is not a measurement of anything.” He also writes: “an incorrect number looks the same as a correct one.” I wrote about the word version of this in June, in The Confidence Problem. The number version is worse, because a number looks like something that was measured and created and has real meaning.
The Hacker News thread on his post ran past a hundred comments. Some of it was engineers arguing about various things, but the idea that kept surfacing was to drop the zero-to-100 scale entirely. One commenter, kergonath said that when a simple ok or not-ok isn’t granular enough, “the vast majority of the time it means that you need more axes.”
A better number
Joshi’s answer is to build the score from things you can observe (was a source found, do repeated answers agree) and then check it against what actually happened, over and over. Researchers have found that with certain prompts a model’s self-reported confidence can be well calibrated, though it depends heavily on how you ask. Calibrated numbers are useful, too: a weather forecast is a number, and people plan their week around it.
The forecast works because you don’t need to know why it’s going to rain to grab an umbrella. A decision about a stuck project, a deal, or a risk depends on exactly that. Calibration tells you how often a tool’s 0.85s come true across hundreds of them. Say a tool was 85% sure that the P1 ticket was what had the team stuck. If that turned out wrong, you’d be left guessing whether the team was ever stuck, whether it was a different ticket, or whether the blocker cleared yesterday and nobody noticed.
An executive looking at sixty projects needs a way to sort them, and nobody reads sixty paragraphs, so a score has a real job. The trouble starts when the number is also the thing you decide on. Use the score to pick where to look, then ask for the sentence on the ones you’re about to act on.
That ticket ended up shaping how we built Scout at Cascade. When Scout says someone is stuck, it has to say who, on what, since when, and what would clear it. If it can’t say all four, it says nothing, and when a blocker is too vague to match a ticket, it says that instead of guessing. It slipped when Scout couldn’t check whether a ticket was still open and the sentence people read didn’t mention it, so a ticket we’d already closed looked as live as an open one, until we put the missing check into the sentence itself.
Most of the dashboards I’ve looked at already know more than they show. The confidence interval sits in a tooltip, the sample size is in a footnote, and the “data as of” date is in a field nobody opens. The people in the meeting read the headline number and the sentence next to it, and nothing else exists.
So when a status number shows up in a meeting (a project health score, a risk rating, a deal likelihood), ask for the sentence that goes next to it. A 70% deal likelihood might come back as “the buyer’s champion has been waiting on a security review since the demo, and it moves when their legal team signs off.” Everyone in the room can check that, and if any piece of it is wrong, somebody in the room will know.