Training reasoning models under a fixed token cap teaches them to stop using more, and given room they fall behind their base models
• 6 min read • By Research Desk
Key findings
- After GRPO training with a 4,000-token rollout cap, Qwen3.5-4B scored 35.9% on AIME 2024/2025 at a 4k budget against its base model's 1.1%, but 48.1% against 78.3% when both could use 64k tokens.
- Averaged over three seeds, all six math-trained checkpoints across Qwen, Gemma and Nemotron gained 10.9 to 42.4 points over their base models at the training cap and trailed them by 13.1 to 30.2 points at the largest budget tested.
- Forcing the 4k-trained Qwen model to keep reasoning by appending "Wait" left its 64k AIME accuracy unchanged, so the lost scaling is not just a habit of stopping early.
- Fine-tuning on the model's own naturally short correct answers kept AIME at 83.0% at 64k, while fine-tuning on artificially truncated ones cut it to 40.4%, pointing at length-censored training data rather than the RL objective.
- On GPQA-Diamond and LiveCodeBench the 8k-trained Qwen checkpoint lost only 2.2 and 1.0 points at 64k while its average output fell from 11.15k to 4.56k and from 38.32k to 8.31k tokens.

Reasoning models are sold on test-time scaling: give them more tokens and they solve harder problems. The reinforcement learning with verifiable rewards (RLVR) that builds them usually runs under a fixed rollout cap, because training cost grows steeply with sequence length, and the resulting model is often evaluated at or near that same cap. A preprint posted to arXiv on November 19 by Ilia Zavidnyi, Alexander Kovrigin, Arina Puchkova, Kirill Gelvan and Sergey Titov of JetBrains Research (Zavidnyi is also at the Czech Technical University in Prague), "Budget Internalization: Fixed-Budget RLVR Trades Test-Time Scaling for In-Budget Accuracy", asks what that cap does to the model beyond saving compute. The authors trained Qwen3.5-4B with GRPO on the DeepScaleR math set under 4k and 8k token caps, and Gemma4-E2B and Nemotron-3-Nano-4B under 1k and 2k caps, then repeated the runs on Reasoning Gym's 104 procedurally generated environments. Every cap sat below the length at which the base models usually finish an AIME answer, and a rollout that hit the cap without an answer got zero reward.
The trained models won where they were trained and lost everywhere above it. On the 60 problems of AIME 2024 and 2025, the 4k-trained Qwen scored 35.9% at a 4k budget against 1.1% for its base model, but when both could use 64k tokens it plateaued at 48.1% while the base model reached 78.3%. The 8k-trained checkpoint went from 50.2% against 7.8% at 8k to 53.7% against 78.3% at 64k. Across all six math-trained checkpoints, averaged over three seeds, the gain at the training cap ranged from 10.9 to 42.4 points and the deficit at the largest budget tested from 13.1 to 30.2 points, and all 18 seeded runs of the main comparison reproduced the crossover. The models simply stopped using the extra room: at 64k the base Qwen averaged 27.3k tokens per AIME answer and the trained checkpoints 5.0k and 6.2k. Training on Reasoning Gym instead of math gave the same shape, with Qwen moving from 31.1 points ahead at 4k to 25.0 points behind at 64k.
Two probes suggest this is more than a learned habit of stopping. First, much of the base model's poor showing at small budgets is truncation, not ignorance: cutting its reasoning at the budget and demanding an answer lifted Qwen's AIME accuracy at a roughly 4k-token budget from 0.6% to 9.4%, and GPQA-Diamond from 4.0% to 60.3%. Second, the reverse does not work. Stripping the 4k-trained model's end-of-thinking marker, appending "Wait" and letting it continue, up to four times under a 64k cap, left its 64k AIME accuracy unchanged and added only 2.7 and 5.0 points on GPQA-Diamond and LiveCodeBench v6, while lowering its accuracy at 4k. Stating a budget in the prompt did not give reliable control either. The authors read this as training having compressed the reasoning itself, and they call the effect budget internalization.
It is not always a loss. The behavior carried from math training to GPQA-Diamond and LiveCodeBench v6, but the damage there was much smaller. At 64k the 8k-trained Qwen trailed its base model by 2.2 points on GPQA and 1.0 on LiveCodeBench while its average output fell from 11.15k to 4.56k tokens and from 38.32k to 8.31k. Nemotron's 2k-trained checkpoint beat its base model on GPQA by 4.0 points at 64k. The authors' reading is that on these tasks the base models often overthink, so a capped model that reaches about the same accuracy with far fewer tokens is a genuine efficiency gain. On AIME, where longer reasoning actually solves more problems, the same compression costs accuracy.
The ablations, run on the 8k Qwen setup and scored over 16 samples per problem, locate the cause in what the training data rewards rather than in any one term of the GRPO loss. Standard training reached 50.4% at 8k and 56.5% at 64k, against the base model's 9.2% and 78.8%. Switching to Dr. GRPO's length normalization changed almost nothing (55.6% at 64k), a soft overlength penalty made it worse (50.0%) and masking truncated rollouts out of the loss helped little (59.7%). Training on the easier GSM8K, where far fewer rollouts hit the cap, kept 72.0% at 64k but gained much less at 8k. A stronger KL anchor to the base model (β = 0.01) kept 70.1% at 64k but managed only 32.4% at 8k, and closing truncated rollouts with a forced answer kept 77.2% at 64k while scoring 7.0% at 8k, below the base model. The cleanest test used supervised fine-tuning with no RL at all: training on the model's own correct answers that finished naturally within 4k tokens left 83.0% at 64k, while training on correct answers produced by cutting longer traces at 4k and forcing an answer dropped it to 40.4%. No intervention kept both the in-budget gain and the base model's scaling.
The limits are clear in the paper. The models have four to five billion active parameters, most experiments train on math, and only the main DeepScaleR and AIME comparison uses three training seeds; every ablation is a single run. AIME has 60 problems, and the 95% confidence intervals in the ablation table reach about 11 points on either side, so small differences between variants are not meaningful, though the main crossovers are far larger than that. The scaling curves come from cutting one 64k-token generation per checkpoint at each budget, not from separate runs at each budget. Only GRPO and its variants were tested, and the authors say their behavioral measurements cannot tell whether a model represents the cap explicitly or has just shifted toward shorter answers. The paper is an unreviewed preprint; the authors have released their checkpoints, including the ablations, on Hugging Face.
For NeuroGenesys the result is a useful counterweight to our own recurrent-depth test-time compute work, where a small recurrent network trained at up to 16 iterations kept improving when run for 64 on harder graphs. There, extra iterations kept refining the same answer; this paper shows that the far more common recipe, RL on a language model under a token cap, can train the same axis away, and that a single-budget evaluation will not show it. It also reads like a quiet form of forgetting: no new domain arrived, yet the trained model ended 30 points below what its base model could do at 64k tokens, and every dial that preserved that capability, such as a KL anchor to the base model, gave up most of the in-budget gain, the trade-off our continual-learning work keeps meeting. As with our note that more thinking is not more evidence, the practical rule is to report accuracy and output length across several budgets before claiming that a training run made a model better. Whether training across several caps or conditioning on an explicit budget keeps both gains is the open question the authors leave, and the one we would want answered first.
Sources
- Budget Internalization: Fixed-Budget RLVR Trades Test-Time Scaling for In-Budget Accuracy — Zavidnyi et al., arXiv 2026
- Budget Internalization: Fixed-Budget RLVR Trades Test-Time Scaling for In-Budget Accuracy (v1) — Zavidnyi et al., arXiv 2026 (full text, HTML)
Related research
- Research Desk • Machine LearningMore thinking isn't more evidence: test-time compute meets correlated error.Sampling a reasoning model sixteen times does not give you sixteen opinions. New preprints put numbers on an old statistical warning.
- Research Desk • Field WatchWhen a reward pays a verdict it cannot check, RL honesty becomes a race against bluffingA new preprint predicts, before training starts, whether reinforcement learning with a checker will teach a model to bluff: honesty survives when the model's success rate times the reward for a checked answer beats the reward for an unchecked one. Across Datalog, code and math abstention, no run above that line collapsed, a checker let runs just below it recover, and paying too much for the honest verdict made two code models call half of correct programs buggy.
- Publication • Test-Time ComputeRecurrent depth as test-time compute: parameter-efficient, extrapolating, and bounded by contractivity
