Majority vote, self-consistency and correlated judges

Asking several models — or one model several times — and taking a vote is the cheapest defence against a bad verifier. This lesson works out when voting helps, why it often helps far less than the head-count suggests, and how two measurable numbers predict the ceiling.

The jury theorem and its assumption

Condorcet's jury theorem (1785) is the classical case for voting: if each of $N$ voters is independently correct with probability $q > \tfrac12$ on a binary question, the probability that the majority is correct rises towards 1 as $N$ grows.^[1] With $q = 0.7$, three independent voters are right as a majority with probability $0.7^3 + 3(0.7^2)(0.3) = 0.784$; eleven are right about 92% of the time.

Self-consistency is this idea applied to one model: sample several chains of thought and return the most common final answer. Wang et al. (2022) reported gains of 17.9 points on GSM8K over greedy chain-of-thought decoding.^[2] Panels of LLM judges, reward-model ensembles and multi-agent debate all lean on the same intuition.

The word doing all the work is independently. Ladha (1992) extended the analysis to correlated votes,^[3] and Kaniovski (2010) gave exact results for homogeneous juries, in which positive correlation lowers the jury's competence and negative correlation raises it.^[4] Language models trained on overlapping data, with shared architectures, prompts and contexts, are not independent voters.

How correlated are model errors?

Measured: highly. Kim et al. (2025), studying more than 350 LLMs, found that on one leaderboard dataset two models agree on the same wrong answer on about 60% of the questions where both are wrong, and that larger, more accurate models are the most correlated, even across providers.^[5] Goel et al. (2025) introduced a chance-adjusted error-overlap metric and found that LLM judges score models more favourably the more similar those models are to the judge, and that mistakes of frontier models have grown more similar as capabilities rose.^[6] In May 2026, Kohli reported that a nine-judge LLM panel on natural-language-inference data carried roughly two independent votes' worth of information; the best single judge matched or beat the full panel, and items on which all nine agreed were still wrong 9.1% of the time.^[7]

Judges also carry systematic biases that act like shared errors. Zheng et al. (2023) documented position, verbosity and self-enhancement biases in LLM-as-judge setups, while finding that strong judges agreed with human preferences over 80% of the time.^[8] Panickssery et al. (2024) showed that models recognise and prefer their own generations, with self-preference strength tracking self-recognition ability.^[9]

Two numbers that predict the ceiling

A simple model captures the effect. Suppose each item is either "poisoned" — every critic in the panel fails on it together — with probability $s$, or clean, in which case critics err independently. Then majority accuracy climbs with $N$ but saturates at $1 - s$: no number of votes rescues a shared failure.

Two observables recover $s$. Let $e$ be the per-critic error rate and $P_{\text{both}}$ the probability that two critics are both wrong on the same item. Then the error correlation and shared component are

and the panel's effective size follows the Kish / Clemen–Winkler form

which tends to $1/\rho$ however many critics you add.^[10] The lab's correlated-critique note recovers $s$ to within $\pm 0.001$ in simulation, and — in its end-to-end run of 8,370 isolated model calls — predicted a real panel's majority accuracy at 0.841 against an observed 0.842.^[11]

Worked example. Critics err 30% of the time ($e = 0.30$) and two critics are both wrong on 15% of items ($P_{\text{both}} = 0.15$; independence would give $0.09$).

• $\rho = (0.15 - 0.09)/(0.30 \times 0.70) = 0.06/0.21 \approx 0.286$
• $s = 0.06/(1 + 0.15 - 0.60) = 0.06/0.55 \approx 0.109$, so the majority ceiling is $1 - s \approx 0.891$
• Five critics give $n_{\text{eff}} = 5/(1 + 4 \times 0.286) \approx 2.33$

If the target is 90% accuracy, it lies above the ceiling: no number of these critics reaches it. The next unit of budget should buy a decorrelated channel — a different model, a different context, a different kind of check — not a sixth copy.

Two scope limits apply. The ceiling formula assumes a homogeneous panel and simple majority; the lab's own soundness flag records that the strong form "any $\rho > 0$ caps accuracy" is false in general, citing Kaniovski's exact results for heterogeneous juries and supermajority rules.^[11] And design advice depends on $N$: in the lab's simulation a more accurate but correlated panel beat a less accurate independent one at $N = 3$, before the independent panel overtook it by $N = 7$.

What makes a critic share the generator's errors

The lab ran a small, deliberately stark pilot. A generator was given a false "company convention" for computing percentage changes, and followed it on 12 of 12 affected items. A critic of the same model with the same context missed all 12 planted errors, in both replicates; the same model with a fresh context, and a different model, caught all 12.^[11] A critic that shares the generator's context shares its standard of correctness. This replicates a phenomenon already reported in the literature rather than discovering it, and the note says so.

The larger follow-up refined the mechanism. The lever was the prompt scaffolding, not access to the reasoning trace: a critic that read the full reasoning but not the poisoned memo caught nearly everything. On natural, model-induced errors, however, fresh-context critics barely helped, and on GSM-Hard the fresh critic's discrimination went negative — it rejected 54% of correct answers, so its low miss rate was a rejection reflex rather than detection.^[11] The methodological rule that follows: a critic study that reports catch rate without false-alarm rate can get the sign backwards.

Voting under adversarial pressure

Voting has a distinctive strength and a distinctive weakness relative to argmax selection.

Strength: counting resists construction. When exploits are built to outscore the field, argmax collapses but plurality among high scorers does not: in the lab's experiment, Best-of-8192 accuracy fell to 0.000 while filter-then-vote held at 0.999.^[12]

Weakness: shared errors win pluralities. If wrong answers concentrate on one value, that value can outvote the truth. With base rate $p$ of correct candidates and a shared exploit taking fraction $q$ of the wrong ones, correct votes number about $pN$ and exploit votes about $(1-p)qN$, so the exploit wins iff

At $p = 0.10$ the threshold is $0.111$, and the lab's simulated crossover fell between 0.11 and 0.15.^[13] The lab later corrected its own shipped guidance: at low solve rates the vote is often lost to neither the right answer nor an exploit but to a dominant ordinary mistake — a plausible wrong answer many samples share — which the two-way threshold does not model.^[12]

Against an adaptive generator that optimises around the panel, decorrelation helps less. In a toy coverage game, a fully decorrelated panel missed 3.2% of static flaws but 55.9% of adaptive ones; a channel reading evidence outside the generator's control was about eight times better than an equal-budget extra critic.^[14] The literature is split on how far ensembles go: Coste et al. (2024) found conservative reward-model ensembles eliminated overoptimisation for Best-of-N in their setting,^[15] while Eisenstein et al. (2024) found ensembles mitigate but do not eliminate reward hacking because members share error patterns.^[16]

Try it

Open /research/tools/verifier-panel. The defaults ($e = 0.30$, $P_{\text{both}} = 0.15$, $N = 5$, target 0.9) reproduce the worked example: $\rho \approx 0.29$, ceiling $\approx 0.89$, $n_{\text{eff}} \approx 2.3$, and the target flagged as above the ceiling. Lower $P_{\text{both}}$ to 0.10 and watch the ceiling rise above 0.9; then raise $N$ to 50 and note how little $n_{\text{eff}}$ moves. To measure your own panel, score two critics on the same labelled items and count how often both are wrong.

Key takeaways

• Voting's guarantees depend on independence, and LLM critics, judges and samples are strongly correlated.
• Per-critic error $e$ and pairwise co-error $P_{\text{both}}$ give $\rho$, the shared component $s$, the majority ceiling $1 - s$ and $n_{\text{eff}} \to 1/\rho$ (for homogeneous, simple-majority panels).
• Past the ceiling, buy decorrelation — different model, context or kind of evidence — not more votes.
• Counting resists constructed exploits that defeat argmax, but loses to shared errors once $q > p/(1-p)$, and to dominant ordinary mistakes at low solve rates.
• Report false-alarm rates alongside catch rates for any critic.

Sources