Temperature and Top-p, Explained
Why the same prompt gives different answers, and how the two main sampling knobs actually reshape the model's choices.
On this page
Send the same prompt twice, get two different answers. This is not instability — it is a deliberate randomness step, and you control it.
At each step, the model outputs a probability distribution over its whole vocabulary. Something has to pick one token from that distribution. How it picks is where temperature and top-p live.
Why not always take the most likely token?
Taking the top choice every time is called greedy decoding, and it produces worse text than you would expect.
It repeats itself, falls into loops, and reaches for the safest, blandest phrasing available. The reason is that natural language is not the most probable sequence of words — real writing is full of locally surprising choices. Optimizing every step for likelihood produces prose that reads like it is hedging.
So randomness gets injected on purpose.
Temperature
Temperature reshapes the distribution before sampling. It divides the model’s raw scores by a number, then renormalizes.
Suppose the model’s top four candidates are 60%, 25%, 10%, 5%.
Low temperature (≈0.2) exaggerates the gaps. That distribution becomes something like 95% / 4% / 1% / ~0%. The favorite almost always wins. Output is focused and repeatable.
Temperature 1.0 leaves the distribution as the model produced it.
High temperature (≈1.5) flattens the gaps — perhaps 40% / 28% / 19% / 13%. Unlikely tokens get real chances. Output is varied, and past a point, incoherent.
The intuition: temperature controls how much the model’s own preference is respected. Low means “follow your training closely.” High means “surprise me.”
Temperature 0 is effectively greedy decoding. Useful when you want the same answer every time.
Top-p
Top-p attacks the same problem differently. Instead of reshaping probabilities, it truncates the candidate list.
Sort tokens by probability, accumulate from the top, and stop once the running total passes p. Sample only from that set.
With top_p = 0.9 and a confident distribution (95% / 3% / 1% / …), the first token alone crosses 0.9 — the set has one member, and output is effectively deterministic. With a flat distribution (12% / 11% / 10% / 9% / …), it takes many tokens to reach 0.9, so the pool is wide.
That adaptivity is the appeal. Top-p lets the model’s own confidence set the width of the choice. When the next token is obvious, it stays obvious. When genuinely open, variety is allowed.
Compare with top_k, which always takes a fixed number of candidates regardless of confidence — cruder, because 40 candidates is far too many when one token is 99% likely.
Using them together
Most APIs expose both. Adjusting both at once makes the effect hard to reason about, so the common advice is to pick one and leave the other at its default.
Practical starting points:
| Task | Setting | Why |
|---|---|---|
| Extraction, classification, structured output | temperature 0 | You want one correct answer, reproducibly |
| Code | temperature 0–0.3 | Correctness matters more than variety |
| Summaries, factual answers | temperature 0.3–0.7 | Some fluency, still grounded |
| Conversational assistants | ~0.7–1.0 | Repetitive replies feel robotic |
| Brainstorming, creative drafts | 1.0+ | Variety is the point |
What temperature does not do
Two frequent misunderstandings worth clearing up.
Temperature 0 does not mean accurate. It means the model’s favorite token wins every time. If its favorite is wrong, you get the same wrong answer reliably. Temperature controls variance, never correctness.
High temperature does not create creativity. It widens the sampling pool. Genuine novelty comes from the prompt and the model’s capability; temperature only changes how far down the ranked list you are willing to reach.
Reproducibility
Temperature 0 gets you close to deterministic but not always exactly. Floating-point non-determinism in parallel hardware, batching effects, and silent model updates can all shift output. Some providers offer a seed parameter to help, though guarantees vary. If exact reproducibility matters, log the outputs rather than assuming you can regenerate them.
What to remember
- Randomness is deliberate: greedy decoding produces repetitive, flat text.
- Temperature reshapes the distribution — low sharpens, high flattens.
- Top-p truncates the candidate pool adaptively based on the model’s own confidence.
- Change one, not both. Temperature 0 for extraction and code; higher for conversation and ideation.
- Neither knob affects accuracy — only variance.
Next: Base Models vs Chat Models — why raw prediction engines do not behave like assistants.