Reasoning Models

Why This Matters

Since late 2024, the frontier of AI capability has shifted from "bigger models" to "models that think longer." OpenAI's o-series, Anthropic's extended thinking, DeepSeek-R1, and Gemini's thinking modes all follow the same recipe. Choosing between a fast standard model and a slow reasoning model, and knowing when the 10× cost is worth it, is now a core engineering decision. If you don't understand the mechanics, you will make one of two expensive mistakes: routing everything to a reasoning tier and watching your inference bill multiply, or routing nothing to it and shipping an agent that fails on exactly the multi-step problems your users care about. A team that routes 100% of a 1M-request/day workload to a reasoning model when 5% would suffice can easily burn 10–20× the necessary spend.

Everyday Analogy

Ask a colleague "what's 17 × 24?" and demand an instant answer, and they will guess and often miss. Give them scratch paper and thirty seconds, and they will get it right. Standard LLMs answer instantly, in one pass. Reasoning models get the scratch paper: they write out intermediate steps, check themselves, backtrack, and only then answer. The analogy maps precisely: the scratch paper is the thinking-token stream, the thirty seconds is added latency, and the paper itself costs money, because you are billed for every token of scratch work, even when the product hides it from you.

From Chain-of-Thought to Trained Reasoning

The idea evolved in three stages, and knowing the stages explains why the models behave the way they do.

1. Prompted chain-of-thought (2022). Wei et al. showed that simply adding worked examples with intermediate steps, and later just the phrase "let's think step by step," made models markedly better at math and logic. On the GSM8K grade-school math benchmark, chain-of-thought prompting lifted PaLM 540B from roughly 18% to 57% solve rate in the original paper. This was the first hard evidence that intermediate tokens act as working memory: a transformer does a fixed amount of computation per token (see AI-012), so more tokens literally means more computation applied to the problem.
2. Test-time compute as a scaling axis (2024). Researchers (notably Snell et al., "Scaling LLM Test-Time Compute") showed that for reasoning-heavy tasks, letting a smaller model generate longer, sample multiple solution paths, and pick the best one can beat a much larger model answering once. Compute spent at inference became a scaling axis alongside training compute: you can buy accuracy per query instead of per training run.
3. Trained reasoning models (late 2024 onward). Instead of hoping a prompt triggers good reasoning, models are trained with reinforcement learning to produce useful internal reasoning: exploring approaches, catching their own mistakes, and verifying before answering. OpenAI's o1 announcement showed accuracy on competition math climbing smoothly as both RL training compute and test-time thinking length increased. DeepSeek-R1 then showed the behavior can emerge from RL with automatically checkable rewards (did the math answer match, did the code pass the tests) with no human step-by-step labels required. During training, R1-Zero spontaneously developed longer chains, self-correction, and "wait, let me reconsider" moments; the researchers called one such transcript an "aha moment."

The mechanism worth internalizing: nothing magical happens inside a thinking phase. The model is still doing next-token prediction (see AI-009). What changed is that RL selected for token sequences that function as search (propose, check, backtrack) because those sequences led to verifiable correct answers.

How a Reasoning Model Answers

A reasoning model splits its output into a long thinking phase (hundreds to tens of thousands of tokens of exploration, often hidden or summarized in products) followed by the final answer. You are billed for both. That single fact drives all the practical trade-offs:

Three engineering consequences follow directly:

• Context consumption. Thinking tokens occupy the output budget. A model with a 64K output limit that spends 30K tokens thinking has 34K left for the answer. In agent loops, discarded thinking from previous turns usually does not persist, but the current turn's thinking counts against limits.
• Tunable depth. Providers expose knobs: Anthropic's extended thinking takes an explicit thinking-token budget per request, while OpenAI's reasoning models accept a low/medium/high reasoning-effort setting. Reasoning depth is a request-time parameter, not a fixed model property, so the same model can serve both your cheap tier and your deep tier.
• Faithfulness limits. The visible reasoning is useful for debugging but is not a guaranteed record of the model's actual computation. Anthropic's own research on chain-of-thought faithfulness found models frequently omit influences (like hints embedded in the prompt) from their stated reasoning. Treat thinking text as evidence, not testimony.

The Menu of Test-Time Compute Strategies

"Think longer" is actually a family of techniques, and you can apply several of them to any model, not just trained reasoning models:

• Longer single chain. One sample, more intermediate steps. Cheapest to implement; this is what a thinking budget controls. Works because each generated token buys another fixed slice of transformer computation.
• Self-consistency / majority voting. Sample the same problem N times at nonzero temperature and take the most common final answer. Wang et al.'s self-consistency paper showed large gains on GSM8K just from sampling ~40 chains and voting, with no model change at all. Cost scales linearly with N.
• Best-of-N with a verifier. Generate N candidates and have a separate checker (a reward model, a unit test, a compiler) pick the best. When a cheap, reliable verifier exists (code that either passes tests or doesn't), this is often the highest accuracy per dollar.
• Search over steps. Tree-of-thoughts-style methods branch at intermediate steps and prune bad branches early, spending compute where the problem is hardest rather than uniformly.

The Snell et al. result that made this a headline: allocating a fixed compute budget optimally at test time let a smaller model outperform a model ~14× larger answering once, on problems where the smaller model had any foothold. The caveat matters too: test-time compute cannot rescue a model on problems entirely beyond it; extra thinking multiplies existing competence, it does not create knowledge.

Trained reasoning models internalize much of this menu: the RL process teaches a single chain to behave like a search: branching ("alternatively, I could…"), verifying ("let me check: 23 × 2 + 12 × 4 = 94 ✓"), and backtracking, so you get search-like accuracy without orchestrating N parallel samples yourself.

How Reasoning Is Trained: The R1 Recipe

DeepSeek-R1's openly published recipe is worth knowing because it demystifies the whole category (see AI-076 for the general RLHF pipeline this extends):

1. Start from a strong base model (DeepSeek-V3, a 671B-total-parameter MoE; see AI-064).
2. Define verifiable rewards. For math: does the boxed final answer match the reference? For code: do the tests pass? There is no reward model guessing at quality, just a deterministic checker. This sidesteps reward hacking on fuzzy criteria.
3. Run large-scale RL (GRPO) where the model generates full reasoning attempts and is rewarded only on outcome correctness plus basic format compliance. Over training, average response length grew on its own, because the model discovered that longer, self-checking chains earn more reward. Nobody told it to backtrack; backtracking got reinforced because it worked.
4. Clean up with SFT. Pure-RL R1-Zero reasoned well but produced messy, language-mixing chains, so the final R1 added supervised fine-tuning stages on curated reasoning data for readability and general helpfulness.
5. Distill. Fine-tuning small open models (7B–70B, Qwen and Llama bases) on ~800K R1-generated reasoning traces transferred much of the ability downward, which is why capable small reasoning models appeared within weeks (see AI-066 on synthetic data).

Two implications for you as a practitioner. First, reasoning quality is strongest exactly where rewards are checkable (math, code, formal logic) and weakest where they aren't (essay quality, taste), so calibrate your expectations by domain. Second, because distillation works, "reasoning" is no longer a frontier-lab monopoly; you can run a distilled 14B reasoning model locally (see AI-065 on quantization for fitting it on your hardware).

Worked Example: Costing a Reasoning Query

Trace the economics with concrete numbers. Suppose a standard model costs $3 per million input tokens and $15 per million output tokens (typical mid-tier frontier pricing), and the reasoning mode of the same model has identical per-token prices; the difference is purely how many tokens get generated.

1. The task: "Find the bug in this 300-line race condition and propose a fix." Prompt ≈ 4,000 input tokens.
2. Standard model: generates a 600-token answer in one pass. Cost = 4,000 × $3/1M + 600 × $15/1M = $0.012 + $0.009 = $0.021. Latency ≈ 8 s. It names a plausible-looking culprit, and on genuinely hard concurrency bugs, one-pass answers are frequently wrong.
3. Reasoning model: spends 12,000 thinking tokens tracing lock acquisition order, considers two hypotheses, rejects one after simulating the interleaving, then writes the same 600-token answer. Cost = 4,000 × $3/1M + 12,600 × $15/1M = $0.012 + $0.189 = $0.201. Latency ≈ 90 s.
4. The ratio: ~10× the cost and ~11× the latency for one query. Whether that is a bargain depends entirely on the alternative: if the standard model's wrong answer costs an engineer an hour of chasing the wrong lock, $0.18 extra is the cheapest hour you will ever buy. If the task was "summarize this PR description," you paid 10× for nothing.
5. At fleet scale: 1M queries/day at $0.021 = $21K/day; at $0.201 = $201K/day. Routing 95% to the standard tier and 5% to reasoning = 950K × $0.021 + 50K × $0.201 ≈ $30K/day, capturing most of the capability uplift for one-seventh the all-reasoning bill.

This is why every serious deployment does the math per route, not per model.

Choosing: A Decision Framework

Ask three questions, in order:

1. Is the answer verifiable or high-stakes-if-wrong? Math, code that must compile, legal or scientific analysis, and multi-step agent plans where step 3 depends on step 1 being right are candidates for reasoning. Subjective or low-stakes text (summaries, chat, tone rewrites) calls for a standard model.
2. Does the task actually require multi-step inference? Extraction, classification, translation, and formatting are single-hop; thinking tokens add cost without accuracy (see the trade-off table above). Debugging, planning, constraint satisfaction, and "explain why these two facts conflict" are multi-hop.
3. Can the user tolerate the latency? Autocomplete and live chat cannot absorb 60-second pauses. Overnight batch analysis, code review, and research assistants can.

Then apply the production pattern:

• Default tier: a fast standard model handles all traffic.
• Escalation trigger: route to reasoning when a classifier flags the request as hard, when the fast model expresses low confidence or fails validation (tests fail, JSON doesn't parse), or when the user explicitly requests deep analysis.
• Budget cap: set an explicit thinking budget per request class, e.g., 4K thinking tokens for routine escalations and 32K for competition-grade problems, so a pathological prompt cannot spend unbounded money.

Many teams find that under 5% of traffic genuinely needs the reasoning tier, and hybrid routing captures most of the benefit at a fraction of the cost.

Failure Modes to Watch

• Overthinking. Reasoning models can generate thousands of tokens agonizing over "what is 2+2 in this context?", burning money and occasionally talking themselves out of a correct first instinct. Budgets and routing mitigate this.
• Confident wrong reasoning. A fluent 8,000-token derivation is persuasive even when step 4 contains the error. Verification (run the code, check the arithmetic) still matters, because reasoning reduces hallucination on logic tasks but does not eliminate it (see AI-060 on hallucinations).
• Latency cliffs in agent loops. An agent that calls a reasoning model ten times sequentially turns 90-second thinking phases into a 15-minute task. Reserve deep thinking for the planning step; use fast modes for mechanical tool calls.
• Budget starvation. Set the thinking budget too low and you get the worst of both worlds: the model burns its 1,024-token allowance mid-derivation, then commits to a half-checked answer at reasoning-tier latency. If you cap budgets, monitor how often the cap is actually hit and raise it for the routes where truncation correlates with wrong answers.
• Prompt habits that fight the model. Old-style "think step by step" scaffolding and few-shot reasoning demonstrations were written for standard models; on trained reasoning models they are usually redundant and sometimes counterproductive, constraining a model that already knows how to structure its search. Provider guides for o-series and extended-thinking models consistently recommend plain, direct prompts: state the problem, the constraints, and the required output format, then let the thinking phase do its job.

Real-World Showcase

• DeepSeek-R1 (January 2025) matched far more expensive reasoning models on math and code benchmarks and was released with open weights under an MIT license, proving the recipe was reproducible and igniting a global race in open reasoning models, including a wave of distilled small reasoning models.
• Claude's extended thinking lets developers set a thinking-token budget per request, so reasoning depth became a tunable API parameter, not a fixed model property, and one model spans the cheap tier and the deep tier.
• Coding agents (Claude Code, Cursor) route planning steps to reasoning-capable modes while using fast modes for mechanical edits, an example of hybrid routing in the tools you may already use.

Common Misconceptions

• "Reasoning models are just bigger models." No: the capability comes from how they spend inference tokens, not parameter count. DeepSeek showed distilled models in the 7B–70B range gaining large reasoning improvements purely from training on reasoning traces.
• "Thinking tokens are free because users don't see them." They are billed as output tokens. A 200-token answer may carry 5,000 thinking tokens, so the invisible part is often 90%+ of the bill.
• "The visible chain of thought is a faithful log of the model's computation." It is useful evidence but not guaranteed to reflect the true internal process; faithfulness research shows models omit real influences.
• "Reasoning models are better at everything, so route all traffic there." On single-hop tasks they match standard models while costing 5–50× more and responding far slower, and they can even overthink simple tasks.
• "Chain-of-thought needs human step-by-step labels." DeepSeek-R1 demonstrated reasoning emerging from RL against automatically checkable rewards, with no human-written reasoning demonstrations.

Try It Yourself

1. Give the same puzzle to a fast model and a reasoning model (or one model with thinking on and off): "A farmer has chickens and rabbits: 35 heads and 94 legs. How many of each?" Compare accuracy, latency, and (if visible) token counts. The answer is 23 chickens and 12 rabbits; check both models' work.
2. Then ask both to summarize a paragraph. Notice the reasoning model is no better, just slower. You have just derived the routing rule for yourself.
3. Take your provider's pricing page and redo the worked example above with real numbers: assume a 4,000-token prompt, a 600-token answer, and 12,000 thinking tokens. Compute the per-query cost of each tier and the daily cost at 100K queries with a 5% escalation rate.

Mini Project

Build a two-tier router in ~50 lines of Python. Write a function that sends each incoming question first to a fast model with the instruction "Answer, then rate your confidence 0–10." If confidence < 7 or the question matches a heuristic (contains "prove," "debug," multi-part numbered constraints), re-ask a reasoning model with a 8K thinking budget. Log per-request cost for both tiers using published token prices. Run it over 20 mixed questions (10 trivia/summarization, 10 math/logic) and report: escalation rate, total cost versus all-reasoning cost, and accuracy per tier. You should see the escalated fraction land near the hard questions and the cost land far below the all-reasoning baseline.

Key Takeaways

• Reasoning models spend inference tokens thinking before answering. Test-time compute is a scaling axis alongside model size, and RL training (not human step labels) is what made it reliable.
• The capability jump is real but domain-specific: math, code, planning, multi-step logic; single-hop tasks see cost without benefit.
• You pay for thinking tokens in both money and latency: a 200-token answer can hide 5,000 tokens of billed thinking, a 10× cost multiplier in the worked example above.
• Reasoning depth is now a tunable request parameter (thinking budgets, effort levels), so design per-route, not per-model.
• Production systems route: cheap models for most traffic, reasoning models for the hard ~5%, with explicit thinking budgets as cost guardrails.
• Visible reasoning is evidence, not testimony; verify outputs independently on high-stakes tasks.