Reading AI results critically

A headline number in an AI paper — "solves 56% of issues", "beats a 14× larger model", "a new law with $R^2 = 0.99$" — is the output of a measurement chain: a benchmark, a selection rule, a judge, a compute budget, a statistical test. This lesson gives a working checklist for auditing that chain, drawn from the failure modes covered in this course and from the NeuroGenesys lab's own published corrections.

1. What exactly was counted?

Coverage or accuracy? pass@$k$ counts a problem as solved if any of $k$ attempts succeeds; pass@1 counts the answer the system would actually ship. Both are legitimate, but they answer different questions. Brown et al.'s jump on SWE-bench Lite from 15.9% to 56% is a coverage result over 250 samples, scored with the benchmark's tests;^[1] it shows what repeated sampling can reach, not what an unattended system picks. See /academy/test-time-compute-and-verification/sampling-and-best-of-n.

Who judged? If a learned verifier, reward model or LLM judge decided correctness or preference, the result inherits that judge's blind spots and biases — position, verbosity and self-preference among them.^[2][3] A judge from the same model family as the system being judged is a correlated instrument; see /academy/test-time-compute-and-verification/majority-vote-and-correlated-judges.

Did the check measure the task, or something correlated with it? Tests that under-specify the task can be passed without solving it. Stroebl et al. (2024) found that gains from resampling against unit tests looked better when the same weak tests served as both verifier and benchmark than when stronger held-out tests were used.^[4] METR reported recent frontier models reward hacking on its own agentic evaluation suites — on one RE-Bench task, in every trajectory o3 produced.^[5]

2. Has the model seen the test?

Public benchmarks leak into training data. Zhang et al. (2024) wrote GSM1k, 1,250 new grade-school maths problems built without language-model assistance to mirror GSM8K, and found accuracy drops of up to 13% for some model families, alongside a positive relationship between a model's likelihood of generating GSM8K examples and its GSM8K–GSM1k gap. Frontier models showed little sign of overfitting.^[6] Liang et al. (2025) asked models to name the buggy file in a SWE-bench Verified issue from the issue text alone, with no repository; one frontier model reached up to 76%, with markedly lower performance on repositories outside the benchmark — a signature of memorisation rather than reasoning.^[7]

The defensive habit: ask whether the evaluation set is newer than the model's training cutoff, whether a fresh held-out variant exists, and whether the paper reports both.

3. Is the leaderboard itself a selection process?

A leaderboard where submitters can test many variants privately and publish only the best is Best-of-N against the leaderboard's noise. Singh et al. (2025) audited Chatbot Arena and reported one provider testing 27 private variants before a release, uneven access to arena data favouring proprietary models, and estimated gains of up to 112% on the arena distribution from additional data access.^[8] This is the optimiser's curse in institutional form: the maximum of many noisy estimates is biased upwards. It is the same mathematics as the verifier lesson, /academy/test-time-compute-and-verification/verifiers-and-exploitability, with the leaderboard as the verifier.

4. Are the error bars real?

Evaluations are experiments on a sample of questions. Miller (2024) argued for reporting the standard error with every eval score, analysing paired per-question differences when comparing two models, and planning sample sizes with a power analysis.^[9]

Worked example. A model scores 78% on a 500-question test. If questions are a sample from a larger population, the standard error is

so a 95% interval is about $78\% \pm 3.6$ points. A rival at 80% on the same set is not distinguishable by unpaired intervals alone; a paired test on per-question outcomes is usually tighter, because the two models succeed and fail on many of the same questions. Resampling the model at nonzero temperature adds a second variance component on top.

5. Was the comparison fair on compute?

"Small model plus search beats big model" is meaningful only at matched compute. Snell et al.'s comparison was FLOPs-matched, and its 14× result held only on problems where the small model already had some success.^[10] Look for the denominator: samples, tokens, wall-clock or FLOPs, and whether the baseline was given the same budget and the same selection rule.

6. Is the claimed regularity a discovery or an identity?

The NeuroGenesys lab publishes the record of its own over-claims, and they make a useful catalogue.

• An $R^2$ that is too good. The lab once reported a cross-domain "law" for verifier error, fit at $R^2 = 0.997$. It was the Gaussian difference identity — Thurstone's 1927 Case V — and the $R^2$ measured simulation noise. It broke by up to 723× under heavy-tailed noise.^[11] The lab's novelty gate turns this into a rule: an $R^2 \approx 0.99$ that holds across unrelated domains should prompt "I have derived an identity" before "I have found a law". Its other checks: does the claim forbid any observation; is the object old enough that the question was already askable; does the relation survive changing microscopic details; and a hostile prior-art sweep, including the citation graph and the sources a paper says it extends.^[12]
• Prior art found late. A two-regime account of verifier failure under adaptive versus static adversaries was presented as apparently unclaimed, then withdrawn on finding Dwork et al. (2015) on adaptive data analysis.^[13][14]
• A superseded draft. The lab's first verifier paper is marked "Superseded draft" in its archive; its successor keeps the measured constants and tool, and explicitly assigns the identity, the coverage scaling and the overoptimisation laws to their original authors.^[15][16]

None of this makes the remaining results wrong. A true claim reached first by someone else is still true; the error is in the attribution and the framing.

7. Could the instrument have shown the effect?

A negative result is only informative if the test could have come out positive. The lab's instrument gate records seven pre-registered predictions that missed on a single day, each because the registered statistic could not have shown the effect: a threshold tighter than the Monte-Carlo noise, a comparison window already on a plateau, two quantities collinear by construction, a run shorter than the slowest relaxation time.^[17] Its proposed remedy is a five-line power step in every pre-registration: floor and ceiling, noise floor, saturation, degrees of freedom, horizon. A miss that fails these is void — worse than a clean refutation, because the question went untested.

The same note found a cure already in the archive: a per-pool regime detector scored 2 of 4 because its signal was smaller than the per-pool noise, while averaging the same statistic over a window of about 50 pools scored 4 of 4.^[17] Pool before you conclude.

8. What would falsify it?

The most useful sentence in a results section names the observation that would overturn the claim. The lab's claims register lists each surviving claim with its evidence and its falsifier, and keeps the dead claims on the same page.^[18] When a paper offers no falsifier, supply one yourself; if you cannot, the claim may be empty rather than wrong.

A one-paragraph checklist

Coverage or accuracy? Which judge, and is it correlated with the system? Could the benchmark be contaminated, and is there a fresh variant? Was the reported model one of many tried? What are the standard errors, and was the comparison paired? Was compute matched? Is the "law" an identity or old result? Could the test have shown the effect? What would falsify the claim?

Key takeaways

• Separate coverage (pass@$k$) from shipped accuracy, and identify the judge behind every number.
• Contamination and private best-of-many submission inflate public benchmarks; prefer fresh held-out variants.
• Report and demand standard errors and paired comparisons; a 2-point gap on 1,000 questions is often noise.
• Treat near-perfect cross-domain fits as likely identities, and check whether a test was powered before reading a miss as evidence.
• Every claim worth keeping should come with a falsifier.

Sources