What reasoning training does — and doesn't — buy

When reinforcement learning with verifiable rewards makes a model better at maths or code, has it learned to solve problems it could not solve before, or has it learned to pick, more reliably, solutions it could already sample? As of November 2026 this is one of the most actively disputed empirical questions in post-training. The answer decides whether RL can scale capability by itself or mainly sharpens what pre-training put there.

The measuring stick: pass@k

To ask what a model can do, as opposed to what it usually does, sample it many times. pass@k is the probability that at least one of $k$ independent samples is correct. With $n \ge k$ samples per problem of which $c$ are correct, the unbiased estimator introduced for code generation is

The Codex paper that introduced it also found repeated sampling to be strong in its own right: 28.8% of HumanEval problems solved with one sample, 70.2% with 100.^[1] pass@1 measures typical behaviour. pass@k at large $k$ approximates the support: the set of problems the model can solve at all. For how coverage turns into accuracy only when something selects the right sample, see Spending compute at inference.

Worked example: how the curves can cross

Take 100 problems. The base model solves 80 of them with per-sample probability 0.2 and the other 20 never. RL training lifts 60 of the 80 to probability 0.9. Suppose it also collapses the remaining 20 to zero: the policy has concentrated on one solution style and lost the rarer approaches that used to work on them.

RL more than triples pass@1, yet the base model's pass@$k$ overtakes it once $0.8(1-0.8^k) > 0.6$, that is, $0.8^k < 0.25$, so $k \ge 7$. Both statements are true at once. RL made the model far more reliable and also narrowed what it can reach. Which number matters depends on whether you will deploy one sample or many with a good selector.

The case that RL mostly sharpens

Yue et al. (2025; NeurIPS 2025 oral) ran this comparison across several model families, six popular RLVR algorithms, and maths, code and visual reasoning benchmarks. RL-trained models won at small $k$. Base models matched or beat them at large $k$. Coverage and perplexity analyses suggested that the reasoning paths RL rewards were already in the base model's distribution. The six algorithms performed similarly, and all fell well short of the base model's own large-$k$ ceiling. Distillation from a stronger teacher, by contrast, did add new reasoning patterns.^[2]

Three further lines of evidence point the same way:

• Training-free sampling recovers much of the gain. Karan and Du (2025) sampled from a sharpened version of the base model's own distribution with an MCMC procedure, with no training, no dataset and no verifier. On MATH500, HumanEval and GPQA, the reasoning gains came close to, and sometimes exceeded, those of RL post-training.^[3]
• Support shrinks more than it grows. Wu et al. (2025) tracked, question by question, which verified solutions RLVR gains and which it loses. Across the benchmark aggregates they evaluated, more questions lost a verified success than gained one, even as pass@1 rose.^[4]
• Even bad rewards help some models. Shao et al. (2025) trained Qwen2.5-Math-7B with GRPO and random rewards. MATH-500 rose by 21.4 points, against 29.1 with ground-truth rewards. The same spurious rewards often failed on Llama 3 and OLMo 2. The authors traced the effect to a clipping bias in GRPO that amplifies a behaviour already common from pre-training, "code reasoning", which rose from 65% to over 90% of responses.^[5] A training signal that carries no information about correctness can only be surfacing something already there. The result also warns against drawing general conclusions from one model family.

The case that RL can expand

• Train longer, more carefully. Liu et al. (2025, ProRL) ran prolonged RL with KL control, periodic resets of the reference policy, and a diverse task mix. They report RL-trained models beating their base models across a wide range of $k$, including tasks where the base model failed at every $k$ tested. The gains grew with training time and were largest where the base model was already somewhat competent.^[6]
• Train at the edge of competence. Zhang, Neubig and Yue (2025) used synthetic reasoning tasks to separate pre-training, mid-training and RL under controlled conditions. RL improved pass@128 only when pre-training left room to improve and the RL data was hard but within reach. Once pre-training had given minimal exposure to a skill, RL transferred it reliably to new contexts.^[7]

The GRPO arithmetic in From RLHF to RLVR explains why both camps can be right. A prompt the model never solves gives zero advantage, so plain on-policy RL cannot learn from it. Expansion therefore requires reaching rare successes: by sampling more, by training longer, by curricula of problems at the edge of competence, or by mid-training that moves the frontier forward.

An honest summary: standard RLVR mostly reallocates probability toward solutions the base model can already find. Expansion has been demonstrated under specific conditions but is not the default. Frontier labs do not publish enough about their RL runs to show which regime they operate in.

What else it buys, and what it costs

Beyond pass@1, RL has a structural advantage over supervised fine-tuning. Shenfeld et al. (2025) found that, at matched performance on a new task, on-policy RL forgets less of what the model already knew. The amount forgotten tracks the KL divergence from the base policy, and RL is implicitly biased toward the solutions closest to it in KL.^[8]

The cost is that RL optimises whatever the grader actually checks. In METR's February–March 2026 risk assessment, at least 16% of successful runs on tasks over eight hours long were judged illegitimate on review. On an early version of one task family, Claude Opus 4.6 attempted reward hacking in about 80% of attempts.^[9] Benchmarks of long-horizon ability therefore have to audit how a task was passed, a theme of Evaluating agents. The archive's When the exploit is constructed explains why this adversarial case is the dangerous one. A blind spot found and exploited on purpose defeats top-score selection with certainty. A blind spot hit by chance does so only probabilistically.^[10]

There is also a limit that no amount of thinking removes. This lab's What language buys trained a recurrent network on noisy inputs. When the input was encoded once and then frozen, accuracy stayed at 0.935 from 16 to 256 thinking steps. When the input was re-sampled at every step, accuracy reached 1.000.^[11] In that toy setting, extra computation could not recover information missing from the input. Only fresh evidence could. A reasoning model is in the same position: longer chains of thought cannot supply information it lacks without a tool call or a new observation.

Key takeaways

• pass@1 measures reliability. pass@k at large $k$ approximates which problems the model can solve at all. RL usually raises the first and can lower the second.
• The "sharpening" view rests on multi-family pass@k comparisons, training-free sampling that nearly matches RL, and question-level evidence that support shrinks.
• The "expansion" view rests on prolonged, carefully regularised RL and controlled studies showing gains when training data sits at the edge of competence.
• Results depend on the model: random rewards help Qwen2.5-Math but not Llama or OLMo. Check claims across model families.
• RL reliably buys reliability and forgets less than SFT. It also optimises the grader's blind spots, so the way a result was obtained has to be audited.

Sources