Making a Model Think Step by Step

Written-out reasoning is not a psychological trick. Each token gets one forward pass, so intermediate steps are genuine additional computation.

Ask a model a multi-step arithmetic question and demand only the answer. It often gets it wrong. Ask the same question and let it work through the steps, and it often gets it right.

This looks like a trick. It is not. It is a direct consequence of how generation works.

The mechanism

Each token gets exactly one forward pass through the network. One pass is a fixed amount of computation — substantial, but finite and constant per token.

So a question requiring five reasoning steps, answered in one token, gets one pass of computation for all five steps. There is nowhere for intermediate work to live.

Now let the model write the steps out. Each step is tokens. Each token gets its own forward pass. And critically, every step is visible as input to subsequent steps — the generated text becomes working memory.

Chain-of-thought converts a single-pass problem into a multi-pass problem. The reasoning is not theater; the tokens are the computation.

What triggers it

Explicit instruction. “Work through this step by step before answering.” Simple and effective.

Structural prompting. Asking for a specific decomposition: identify what is given, state what is asked, then solve.

Demonstration. Few-shot examples that include the working, not just the answer. This is the most controllable version, because you specify the shape of reasoning you want.

Reasoning models. Newer models are trained to generate extended reasoning before answering, without being asked. The mechanism is the same — more tokens, more passes — moved from prompt to training.

Where it helps

Tasks with genuine sequential dependency:

• Arithmetic and unit conversion — each operation needs a place to happen
• Multi-constraint problems — scheduling, logic puzzles, anything where constraints interact
• Debugging — trace execution, then diagnose
• Comparative analysis — evaluate each option before ranking
• Long causal chains — one step at a time is far more reliable than one leap

Where it does not

Simple lookups. “What is the capital of France?” gains nothing from deliberation and costs tokens.

Pure format transformation. Reformatting has no reasoning to decompose.

Missing knowledge. Reasoning cannot recover information the model does not have. It produces elaborate, well-structured wrong answers — arguably worse than a short wrong answer, because the structure is persuasive.

Genuinely hard problems past capability. More steps do not manufacture ability the model lacks.

The uncomfortable caveat

The written reasoning does not necessarily reflect what actually determined the answer.

Research has demonstrated cases where a model reaches a conclusion, then produces reasoning that justifies it, and where changing a hint alters the answer while the stated reasoning does not mention the hint at all. The visible chain can be a plausible-looking narrative rather than a faithful trace.

Two things remain true simultaneously: the extra tokens genuinely improve accuracy, and the stated reasoning is not a reliable audit trail. Treat it as working space that helps, not as an explanation you can trust.

This matters practically. If you need to verify why an answer was reached, checking the stated reasoning is weaker evidence than it appears.

Practical notes

Reasoning before answer, always. Tokens generated after the answer cannot influence it — the answer is already committed. An explanation appended afterward is rationalization, mechanically speaking.

Ask for a clearly delimited final answer. Otherwise extracting it programmatically is fragile. “End with ANSWER: <value>” is enough.

It costs output tokens, which are the expensive kind. Reasoning that triples output length triples that portion of the bill. Worth it when accuracy matters; wasteful on simple tasks.

Self-consistency, when accuracy dominates. Generate several independent reasoning paths at nonzero temperature and take the majority answer. Reliably improves accuracy at several times the cost.

What to remember

• Each token gets one forward pass, so written steps are real additional computation, not performance.
• Triggered by instruction, structure, examples that include working, or training.
• Helps on sequential reasoning; useless on lookups, formatting, or missing knowledge.
• The stated reasoning may not faithfully reflect what drove the answer — it helps accuracy without being an audit trail.
• Reasoning must come before the answer; afterward it cannot influence anything.

Next: Getting JSON Out of an LLM