Logits and Softmax

How a model's raw output scores become a probability distribution over the next token. The softmax function, what temperature does to it, and why the numbers are relative, not absolute.

A model does not emit “Paris.” Its final layer emits a long list of raw numbers — one per token in the vocabulary — and those numbers can be positive, negative, large, or small. They are not probabilities yet. Turning them into a clean distribution that sums to 1 is the job of one function, softmax, and understanding it demystifies temperature, top-p, and half the confusion about model confidence.

What logits are

The model’s last step produces one score for every token it knows — tens of thousands of numbers. These raw scores are called logits. A logit of 8.2 for ␣Paris and 3.1 for ␣London means Paris is favored, but 8.2 is not “an 8.2% chance” or anything else interpretable on its own. Logits live on an unbounded scale and only mean something relative to each other.

Two facts about logits matter downstream:

• Their absolute size is arbitrary. Adding 100 to every logit changes nothing about the final probabilities.
• Only the gaps between them carry information. A gap of 5 between two tokens means the same thing whether the logits are (8, 3) or (105, 100).

So the raw output is a field of relative preferences. Softmax reads the gaps and produces probabilities.

What softmax does

Softmax takes the vector of logits and returns a vector of probabilities — all positive, all summing to 1. It does this in two moves.

Exponentiate. Raise e to the power of each logit. This makes every value positive and, crucially, amplifies differences. A logit gap of 2 becomes a probability ratio of about 7.4×; a gap of 5 becomes about 148×. Larger logits pull away fast.

Normalize. Divide each exponentiated value by the sum of all of them. Now they add to 1 and form a valid distribution.

The consequence of the exponential step is that softmax is winner-emphasizing. It does not just rank tokens; it sharpens the lead. A logit that is moderately ahead becomes a probability that is dominantly ahead. This is why models often place most of their mass on one or two tokens even when several are plausible.

Why the exponential, specifically

You could imagine simpler ways to turn scores into a distribution — for instance, shift everything positive and divide by the sum. Softmax uses the exponential for reasons that are not arbitrary.

It handles negative logits cleanly (the exponential of anything is positive), it responds smoothly to changes so gradients flow well during training, and it pairs mathematically with cross-entropy loss to give clean, well-behaved updates. That last point is the deep reason: the training objective and softmax are designed as a matched pair. The loss “undoes” the exponential in a way that makes the math simplify.

Temperature: rescaling before softmax

Here is where a control knob enters. Before applying softmax, divide every logit by a number T called temperature.

• T below 1 stretches the gaps between logits. Softmax then sharpens even harder — the top token’s probability climbs toward 1, and output becomes more deterministic.
• T above 1 compresses the gaps. Probabilities flatten toward uniform, and rarer tokens get a real chance.
• T = 1 leaves the logits untouched — the model’s native distribution.

Temperature does not change what the model believes is likely; it changes how decisively that belief is turned into a bet. At T approaching 0, softmax collapses to always picking the single highest logit, which is greedy selection. The mechanics of picking from the resulting distribution — temperature alongside top-p and top-k — are covered in Temperature and Top-p.

Probabilities are relative, not truth

A common misread: a token at 0.97 means the model is 97% sure it is correct. It does not. The 0.97 is entirely a statement about the logit gaps in this forward pass — how much this token’s score beat the others. It reflects pattern strength, not verified fact.

A model can place 0.99 on a confidently wrong answer. The distribution is sharp because the training patterns pointed hard one way, not because anything was checked against the world. This is a mechanical reason fluent output can still be false: softmax faithfully reports the model’s internal preference, and that preference is a prediction, not a truth value.

What to remember

• The model’s raw output is logits — unbounded scores, one per vocabulary token, meaningful only relative to each other.
• Softmax exponentiates and normalizes them into a probability distribution summing to 1, and the exponential step sharpens the lead.
• Only logit gaps matter; adding a constant to all logits changes nothing.
• Temperature divides logits before softmax: below 1 sharpens, above 1 flattens, near 0 becomes greedy.
• A high probability reflects pattern strength in this pass, not confidence in truth.

Next: Temperature and Top-p — how a token is actually chosen from the distribution softmax produces.