More thinking isn't more evidence: test-time compute meets correlated error.

Test-time scaling rests on an appealing idea: if a model can think longer, or answer many times and vote, its answer should improve. Often it does. A June preprint from McFadyen and colleagues, testing up to 12 frontier models on seven benchmarks, argues that fixed-budget evaluations can increasingly understate frontier capability, and that benchmark scores should be reported as a function of inference compute. More compute buys real performance. The open question is what kind of information that compute produces.

A preprint posted on October 16 takes direct aim at the usual assumption. In Not All Test-Time Compute Is Evidence, Ming Shen and Tianjun Yao argue that extra computation from the same model does not behave like independent evidence, because its errors are correlated across repeated samples, across reasoning depth and in the model's evaluation of its own work. Their remedy, DIA, watches for disagreement between an intermediate answer and the final one, and only then sends the question to a panel of different models that solve it independently. They report that DIA beats full chain-of-thought in 12 of 14 settings across seven benchmarks and two base models, including a 7.00% gain on MuSiQue, while activating on about 17% of examples and using up to 47.2% fewer tokens than a multi-agent baseline.

The statistics behind this are old. For n sources with average pairwise error correlation ρ, the effective number of independent sources is n / (1 + (n − 1)ρ), which levels off at 1/ρ however large n grows. At ρ = 0.5, sixteen samples are worth fewer than two independent ones. We reached the same conclusion in our correlated-critique work, where our literature check found the core framing to be largely classical ensemble theory, going back to Clemen and Winkler in 1985, restated for language models. What is new is the setting. Reasoning models make the correlation easy to overlook, because their repeated attempts look so much like deliberation.

If decorrelation is the lever, then the useful questions change. Lifshitz and colleagues pointed this way in 2025 with multi-agent verification, treating the number of verifiers as its own scaling axis. Dughmi and colleagues add the economics in a May theory paper on search with costly verification, where checking a candidate is expensive enough that the order of checks matters. DIA's 17% activation rate belongs in the same category: spend independent compute only where a cheap signal says it is needed.

A paper from November 4 attacks the waste from the other end. Routes of Reasoning targets overthinking, where a model keeps verifying after it already has the answer. By editing expert router scores in mixture-of-experts models at inference, without any training, the authors report a median 30.4% cut in generated tokens across four models and four benchmarks, with accuracy never falling by more than 0.5 percentage points. If much of a model's re-checking adds little, that fits the correlation story, though the paper tests a different question.

These results need the usual caution. All are preprints, the numbers above come from their abstracts, and none has been independently replicated. Verification will have a visible presence at NeurIPS 2026 next month, in workshop titles such as "Who Verifies the Agents?" and "Can We Trust the Judge?". For anyone building on reasoning models today, the practical advice is already clear: before adding samples, ask how correlated they are.