The Output Projection: From Hidden State to Vocabulary Logits

The final step of a transformer turns one hidden vector into a score for every token in the vocabulary. How that projection works, why it is the largest single matrix, and why models tie it to the embeddings.

After the last transformer layer, a token is still just a vector — a few thousand numbers with no obvious connection to words. One step remains: converting that vector into a score for every possible next token. This is the output projection, sometimes called the language modeling head, and it is where the model’s internal representation finally meets the vocabulary. It is also, by parameter count, often the single largest matrix in the model.

The final vector, and the problem

The final layer outputs one hidden state per position — a vector of the model’s internal width, say 4,096 numbers. This vector is dense and abstract: it encodes everything the model has assembled about what should come next, but not in any form you can read.

What generation needs is different: one number for each token in the vocabulary, expressing how strongly the model favors it. If the vocabulary has 50,000 tokens, the model needs 50,000 scores. So the task is a shape change — from a 4,096-dimension hidden state to a 50,000-length score vector. That is exactly what a single matrix multiplication does.

What the projection computes

The output projection is one large weight matrix with shape (hidden dimension × vocabulary size). Multiply the final hidden state by it and you get one score per vocabulary token. Those scores are the logits.

The mechanism is worth seeing concretely. Each column of the matrix is a learned vector, one per vocabulary token, living in the same space as the hidden state. Computing a token’s logit is a dot product between the hidden state and that token’s column — a similarity measurement. A token whose column points in a similar direction to the hidden state gets a high score; a token pointing elsewhere gets a low one.

So the output projection is doing a nearest-neighbor comparison in disguise: which vocabulary directions best match the direction the model has settled on? The best matches get the highest logits. From there, softmax turns those logits into probabilities and generation proceeds.

Only the last position, during generation

A subtlety about efficiency. The stack computes a hidden state for every position, but during generation the model only needs the next token — so it only needs to project the last position’s hidden state. The projection is applied to one vector, producing one distribution.

During training it is the opposite: the projection runs on every position at once, because teacher forcing grades a prediction at every position in parallel. Same matrix, applied to one vector at generation time and to the whole sequence at training time. This asymmetry is one reason training a sequence and generating it have very different compute profiles even though the weights are identical.

Why it is so large

Notice the shape: hidden dimension times vocabulary size. With a 4,096 hidden dimension and a 50,000-token vocabulary, that is over 200 million parameters in one matrix — frequently the biggest single tensor in the model, rivaling or exceeding any individual feed-forward layer.

This is also why vocabulary size is a real architectural cost, not a free parameter. Every additional token adds a full column to this matrix and a corresponding row to the input embeddings. A larger vocabulary means shorter sequences (fewer tokens per text) but a heavier output projection and softmax. That trade-off is one of the quiet balancing acts in model design.

Weight tying

Here is an elegant move most models make. The input embedding matrix maps tokens into vectors; the output projection maps vectors back to tokens. Both matrices have the same shape — vocabulary size by hidden dimension — just used in opposite directions.

Weight tying uses the same matrix for both. The vector that represents a token on the way in is the same vector used to score it on the way out. This roughly halves the parameters spent on vocabulary handling, and it has a satisfying logic: a token’s identity should be one thing, whether the model is reading it or predicting it. It also tends to improve quality, because the input and output representations of each token are forced to stay consistent. Not every model ties these weights, but many do, and it is a clean example of architecture reflecting a conceptual symmetry.

What to remember

• The output projection turns the final hidden state into one logit per vocabulary token via a single matrix multiplication.
• Each vocabulary token has a learned column; its logit is the dot product with the hidden state — a similarity score.
• During generation only the last position is projected; during training every position is, in parallel.
• The matrix is hidden-dimension × vocabulary-size, often the largest single tensor in the model, which makes vocabulary size a genuine cost.
• Weight tying reuses the input embedding matrix for the output projection, halving vocabulary parameters and keeping each token’s representation consistent both ways.

Next: Encoder, Decoder, and Encoder-Decoder Architectures — three ways to arrange the transformer stack.