The Novelty Problem: Why AI Ideas Don't Survive First Contact
Back in January of last year I wrote a post called LLMs and Creativity where I said, more or less, that these things aren’t creative. When I’m stuck on a product I’ll ask an LLM for ten ways to position it, or ten features that would make somebody pay for it, and it hands me ten in about a second. What comes back is probability, not invention. Things that are likely to go together. That’s still roughly where I land, with one large caveat I did not have in January.
I’ve been making some version of that argument for a long time. In December of 2007 I wrote a post called Innovation needs Execution , which was mostly me agreeing with somebody else that the ideas are the cheap part and the delivery is where things actually get decided. Nineteen years and the observation has not changed much.
Chenglei Si, Diyi Yang and Tatsunori Hashimoto at Stanford ran the experiment that says I’m wrong , the one I would have wanted somebody to run. They recruited more than a hundred expert NLP researchers, had them write research ideas, had an LLM ideation system write research ideas, and then had 79 expert reviewers blind-score all of it. Nearly three hundred reviews, collected over the course of a year.
The LLM ideas scored higher on novelty. 5.64 against 4.84 on a ten-point scale, and the gap was statistically significant. Human ideas came out slightly ahead on feasibility, but on the thing I would have bet the house on, the machine won.
That is a well-built study and it says the opposite of what I wrote. The novelty number is the one that gets quoted, and there is a second half that mostly doesn’t.
The other half of the evidence arrived in June of last year, from the same three authors who ran the first study.
They took the ideas and had somebody actually build them . Forty-three researchers, more than a hundred hours each, turning both the human and the LLM ideas into real papers, which were then reviewed blind by experts who didn’t know which was which.
The LLM ideas fell apart. Novelty dropped 1.049, excitement 1.760, effectiveness 1.879. The human ideas moved 0.010, 0.078 and 0.052: they didn’t move at all. On many of the metrics the ranking flipped outright and the human ideas came out ahead.
Novelty at the idea stage was measuring unfamiliarity. Only the finished work told the two apart.
I’d like to claim I called it. I didn’t, and being wrong here costs me more than being right would have.
I thought the ideas were obviously derivative. That was the whole basis of the position, that you could look at what came out of the machine and see the recombination sitting there in plain sight. But 79 expert reviewers looked at those same ideas and rated them more novel than what their own colleagues produced, and nobody caught anything until somebody spent a hundred hours finding out. I was right about where the ideas end up and wrong about how you’d know.
The flattening
The training that makes a model pleasant to talk to is the same training that flattens it. Kirk and his co-authors measured what RLHF does to output diversity and found it cuts substantially when you ask the same question twice, less so across different questions. They ran that on summarisation and instruction following, so anyone stretching it to idea generation is extrapolating, me included.
Si’s team measured the thing itself. To produce the ideas that beat the humans, their agent generated four thousand seed ideas per topic and only two hundred were distinct. Everything else was a near-duplicate of something it had already written, and the non-duplicate rate kept falling as it ran.
The novelty number sits on top of that pile. So does the ten I get back when I ask, except nobody sorted mine. Springboards shipped a model called Flint in April built specifically to fight the flatness, which tells you it isn’t getting patched out of the general models any time soon.
Anil Doshi and Oliver Hauser ran a writing experiment in Science Advances where some writers got story ideas from an LLM. The assisted stories were judged more creative, better written and more enjoyable, with the largest gains going to the less creative writers. They were also more similar to each other.
Everybody feels more creative while the output quietly converges. There’s no vantage point inside a single session from which you can see that happening, so nobody on your team is going to walk into your office and report it. That’s the one I’d worry about if I ran an innovation program.
DeepMind’s FunSearch found genuinely new mathematics , a result on an open problem that had stood for twenty years, published in Nature. Real discovery, and I’m not hedging that.
But look at how it’s built. The LLM proposes, and a separate automated checker runs every proposal and throws out anything wrong or invented, thousands of times over. The model never gets to decide whether its own idea was any good. The checker isn’t the expensive part either: fifteen machines generating, a hundred and fifty cheap processors grading, across five plain servers.
That’s the whole trick, and it works because a math problem can be graded by a machine. A proposed answer is either right or it isn’t, and a computer settles it in a second with no opinion involved. Almost nothing you’d want an idea for behaves that way. Take the automatic grader out and the only thing left that can tell you whether an idea was good is somebody building it and finding out. That is what I was circling in 2007 without having the vocabulary for it.
None of that argues for keeping these tools out of your ideation process. The open question is who does the filtering, and Si’s team checked whether the model could do its own.
They put an LLM in the judge’s seat, benchmarked it against the human reviewers, and got agreement in the low fifties against a coin flip at fifty. Their words: LLMs cannot evaluate ideas reliably yet. When the first author sorted the pile by hand instead, the overall score went from 4.85 to 5.34, and feasibility did not move at all.
Someone will point out that software has its own automatic graders, and that’s fair. Tests, a prototype, a week in front of ten customers. But those tell you whether the thing works. None of them tell you whether it was worth building, and that is the judgment the novelty score was faking.
Producing a hundred ideas was never the hard part of anybody’s innovation program, and nineteen years later the hard part is still finding out which one survives being built.