Why AI Models Find Different Code Problems
I gave the same 1000-line script to three frontier models and asked each for a code analysis. Same file, same question, identical brief. What came back were three different lists. There was an overlap — every model found the obvious things — but each one also had findings no other model had.
My first instinct was to read that as a weakness: if the models can’t agree, I can’t fully trust any of them. By now I see it the other way around. That spread is exactly why several models together find more than the single best model on its own. I want to take apart why.
An LLM doesn’t search code — it reads selectively
The first misconception sits in the word “search”. A classic static analyzer goes through a codebase exhaustively: every rule against every line, deterministic, complete within its rule set. Whatever it can catch, it catches every time, everywhere.
A language model works differently. It doesn’t read the code line by line against a fixed checklist; it directs its attention selectively, driven by what seems relevant in the given context. That’s closer to an experienced reviewer skimming a file and stopping at three spots than to a linter mechanically working through everything. The consequence: what a model finds depends not only on what’s in the code, but also on where it placed its attention in that particular run.
Even the same model finds different things twice
Before comparing different models, it’s worth looking at a single one. Even running the exact same model twice over the same file, I don’t necessarily get the same result.
The reason is sampling. At every token, the model picks from a probability distribution. Temperature controls how far it’s allowed to stray from the most likely path. Above zero, there’s a random element in play — and it propagates. A word chosen differently early on steers the rest of the analysis in another direction. This path dependence means a run that hits a concurrency issue early keeps digging there; a second run of the same model might start with error handling and end up somewhere else.
Spread is not arbitrariness
Path dependence doesn’t mean the findings are random. Real problems stay real problems. It only means the order and focus of a run co-determine which of the existing problems actually surface.
Limited attention prioritizes — and that creates the difference set
The second reason is limited attention. A model’s attention mechanism can’t weigh everything equally; it prioritizes. With 1000 lines, a model can’t relate every line to every other in full depth. It has to decide what to look at.
That explains the pattern cleanly: the overlap of findings — what all models name — is the obvious. Problems so clear they surface under almost any prioritization. The difference set — what only a single model finds — is the subtle. Problems that only become visible when attention happens to land on exactly that spot, by chance or by training. And the subtle bugs are precisely the expensive ones.
Models have emphases
On top of sampling and prioritization comes a third factor: models are shaped differently. Training, fine-tuning and alignment leave models with built-in emphases. One tends to catch security holes, another performance problems, a third logical inconsistencies or missing edge cases.
This bias isn’t per-run noise but a systematic tendency. It’s the reason the difference set is often not just noise but complementary: model A brings the security findings model B misses, and vice versa. Three different emphases together cover more ground than the same one three times.
What the real test showed
I didn’t just reason about this in the abstract; I tested it on a concrete critical script. Same file, different models and settings, each scored by the sum of valid findings.
| Rank | Model / Setup | Σ Findings | Note | | --- | --- | --- | --- | | 1 | Opus 4.8 xHigh DeepThink, Multi-Agent | 15 | strongest list, but expensive | | 2 | Gemini 3.1 Pro High | 14.5 | best single pass, price-performance winner | | 3 | Opus 4.8 Med DeepThink | 13.5 | | | 4 | Sonnet 4.6 | 13 | line anchors, easy to verify | | 5 | Gemini 3.5 Flash High | 12 | a positive surprise | | 6 | Google Flash 3.5 Low | 10.5 | the only false positive (#8) | | 7 | Haiku 4.5 | 10.5 | misses the top bug (#1) |
An eighth run with Opus that had prior contact with the file ran out of competition — the result was contaminated by that prior contact and therefore not comparable.
The point sits in the first column. Half a point separates first from second place. The gap at the top is tiny. The big jump in the table doesn’t come from one model being radically stronger than the next — it comes from orchestration. The multi-agent run leads because it merges several views, not because a single model is superior.
What I’m testing next
Three tests follow from this observation, all concrete.
First, measure determinism. I run the same model several times at temperature zero over the same file. If the results then stay stable, I can separate luck from skill — and see how large the sampling share of the spread really is.
Second, build a ground truth. I deliberately plant known bugs in a script so I know exactly what’s in there. Only then can recall be measured cleanly: what share of the actually present problems gets found? Without that reference I’m only comparing lists against each other, not against the truth.
Third, pit a deliberately diversified ensemble against the best single pass. If spread really is the strength, an orchestration of intentionally differently-shaped models has to beat the strongest single run — and do it reproducibly.
Conclusion
Three models, three lists — that’s not a defect, it’s information. The agreement shows the obvious, the divergence the subtle. Sampling, limited attention and different shaping together ensure no single model covers the whole surface. The real test confirms it: the lead at the top is small, the leverage is in merging. Whether that holds up under clean measurement is what the next round decides — the tests are set, the outcome is open.
The practical takeaway already holds, though: a spread of models beats any single one, which is the whole point of running them as an orchestrated ensemble instead of betting everything on one.
Entdecke mehr
Effort level and deep thinking: two independent axes for AI tasks
Effort scales breadth, deep thinking scales depth. When each setting makes sense — with three clear examples and one rule of thumb.
GlossarEnsemble / Multi-Model Orchestration
Ensemble means combining several deliberately varied LLM runs or models whose findings complement each other. Multi-model orchestration drives these runs via orchestrators with sub-agents, so the union of results is larger than any single run.
LexikonFunction Calling / Tool Use
How an LLM uses tools: define a tool as a schema, the model picks the function and arguments, the result returns to the chat — the basis of every agent.