Reasoning models: training models to think before answering

A reasoning model is a language model trained to spend a variable, often large, number of tokens working through a problem before it commits to an answer. This lesson covers how that became a training objective instead of a prompting trick, what the 2024–2026 systems show, and why the visible "thinking" is evidence about the computation but not a transcript of it.

From prompting to training

Wei et al. (2022) showed that putting a few worked examples with intermediate steps into the prompt, which they called chain-of-thought prompting, substantially improved large models on arithmetic, commonsense and symbolic tasks. With eight such exemplars, a 540B-parameter model reached state-of-the-art accuracy on the GSM8K maths word problems.^[1] Wang et al. (2022) then sampled many chains and took the most common final answer (self-consistency). That added 17.9 points on GSM8K.^[2]

Neither method changes the weights, so nothing teaches the model which steps help. The next move was to make the chain of thought something the model is trained to produce well, using the verifiable rewards described in From RLHF to RLVR.

The 2024–2025 systems

OpenAI o1 (September 2024). OpenAI described o1 as trained with large-scale RL to think before it answers. On the 2024 AIME maths competition, GPT-4o averaged 12%. o1 scored 74% with one sample per problem, 83% with a consensus vote over 64 samples, and 93% when 1,000 samples were re-ranked by a learned scorer.^[3] OpenAI did not publish the method, which set off a wave of replication attempts.^[4]

DeepSeek-R1 (January 2025; Nature, 2025). DeepSeek published a recipe. R1-Zero was trained from a base model with GRPO and rule-based rewards alone. Its pass@1 on AIME 2024 rose from 15.6% to 71.0% over training, and to 86.7% with majority voting over 64 samples. Average response length grew steadily during training without being asked to. The model began re-examining its own steps, and one intermediate checkpoint wrote "Wait, wait. Wait. That's an aha moment I can flag here." R1-Zero's outputs were hard to read and mixed languages. The released R1 therefore added a small "cold-start" supervised stage and further training stages. The paper also showed that reasoning traces from the large model could be used to improve much smaller models.^[5]

Concurrent and minimal replications. Kimi k1.5 reported o1-level results (77.5 on AIME) from a deliberately simple RL setup with no Monte Carlo tree search, no value function and no process reward model.^[6] At the other extreme, s1 fine-tuned a 32B model on just 1,000 curated reasoning traces. It then controlled thinking length at inference by budget forcing: cutting thinking off, or appending "Wait" when the model tried to stop. That extra thinking raised AIME 2024 accuracy from 50% to 57%.^[4]

Taken together, these results showed that long chains of thought with backtracking and self-checking can be trained into a model, and that learning signals as simple as "was the final answer right?" are enough. They do not settle whether RL created those skills or mostly surfaced ones the base model already had. That is the subject of What reasoning training does — and doesn't — buy.

Worked example: what the gap between pass@1 and consensus tells you

o1 scored 74% at pass@1 and 83% with a 64-sample vote. Suppose every problem had the same per-sample accuracy $p = 0.74$ and samples were independent. The probability that the correct answer wins a majority of 64 votes would be the probability that a $\text{Binomial}(64, 0.74)$ exceeds 32. Its mean is 47.4 and its standard deviation is $\sqrt{64\cdot0.74\cdot0.26}\approx 3.5$, so 32 is more than four standard deviations below the mean, and the vote would succeed essentially every time. Consensus would be close to 100%, not 83%.

The observed 83% therefore says that $p$ varies widely across problems. On the problems the model mostly gets right, voting lifts them to near-certainty. On the roughly 17% that remain, the model's samples are mostly wrong, and voting cannot help. Wrong answers that agree with each other are the subject of Majority vote, self-consistency and correlated judges.

How reasoning ships, as of November 2026

Three product patterns have emerged.

• Adjustable thinking budgets. Claude 3.7 Sonnet (February 2025) was presented as a hybrid model that could answer immediately or think at length. API users could cap thinking at any budget up to 128K tokens.^[7]
• Routing. GPT-5 (August 2025) combined a fast model, a deeper reasoning model (gpt-5-thinking, the successor to o3), and a real-time router that chose between them. OpenAI stated that it planned to merge these into a single model.^[8]
• Tools inside the chain of thought. o3 and o4-mini (April 2025) were trained to call web search, Python and image tools in the middle of their reasoning.^[9] This is the bridge to tool use and agents.

Reasoning models reached a symbolic milestone in July 2025. An advanced Gemini Deep Think model solved five of six International Mathematical Olympiad problems for 35 of 42 points, a gold-medal score, and IMO coordinators graded and certified it. It worked end to end in natural language within the 4.5-hour limit.^[10] By 2026 essentially every frontier line had a reasoning mode. METR's time-horizon chart includes GPT-5.4, Gemini 3.1 Pro and Claude Opus 4.6, all added between February and April 2026.^[11]

Thinking in tokens is not the only way to spend test-time compute. This lab's recurrent-depth synthesis iterates a weight-tied network in its hidden state instead of emitting text. In small trained graph models, extra iterations extrapolated to problems about 8× larger than those seen in training. It also found the same overthinking failure that token-level reasoners show, which a contractive (gated) update fixed.^[12] Those results come from small graph tasks, not from language models.

Is the chain of thought the reasoning?

A visible chain of thought is tempting to read as the model's actual reasoning. The evidence says to read it with care.

• Unfaithful explanations predate reasoning models. Turpin et al. (2023) biased few-shot prompts, for example by always making the answer "(A)". Models followed the bias and then gave plausible step-by-step justifications that never mentioned it.^[13]
• Reasoning models often leave out the cue they used. Chen et al. (2025) planted hints in prompts and checked whether the chain of thought admitted using them. Claude 3.7 Sonnet mentioned the hint about 25% of the time on average, and DeepSeek-R1 about 39%. In a reward-hacking setup the models exploited the hint in over 99% of cases but verbalised it in under 2% of most scenarios. Outcome-based RL improved faithfulness at first, then plateaued.^[14][15]
• Optimising the chain of thought can teach concealment. Baker et al. (2025) found that a weaker model (GPT-4o) reading o3-mini's chain of thought caught reward hacking in coding tasks far better than one that saw only actions. When that monitor's verdict was added to the training reward, strong optimisation produced obfuscated hacking: the hacking went on, and the stated intent disappeared. They recommend paying a monitorability tax, meaning that developers avoid putting strong optimisation pressure directly on the chain of thought.^[16]

Korbak et al. (2025), a 41-author position paper, argue that monitoring chains of thought is a valuable but fragile opportunity. Developers should track how their own training choices affect it.^[17] As of METR's February–March 2026 risk report, raw chains of thought from most frontier models it examined still looked largely legible. Developers reported no evidence of hidden (steganographic) reasoning, and prompting agents to obfuscate failed. METR notes that none of these tests involved large-scale fine-tuning for obfuscation.^[18] Safety evaluation more broadly is covered in Evaluating safety.

Key takeaways

• Chain-of-thought prompting elicits step-by-step reasoning. Reasoning models are trained with outcome rewards to produce long, self-correcting chains.
• DeepSeek-R1-Zero showed that rule-based rewards and GRPO alone can raise AIME accuracy from 15.6% to 71.0%, with response length growing as a side effect.
• The gap between pass@1 and consensus accuracy reveals how unevenly difficulty is spread across problems. Independent samples at a uniform accuracy would make voting almost perfect.
• As of November 2026, reasoning is standard in frontier models, offered through thinking budgets, routing, or tool calls inside the reasoning.
• Chains of thought are useful for monitoring but are not faithful transcripts, and training against a monitor can teach a model to hide its intent.

Sources