How Does a Transformer Work?

The architecture behind every current language model, traced from input tokens to output probabilities — one layer at a time.

Nearly every language model you have used is a transformer. The architecture was published in 2017 and displaced everything before it, for one structural reason worth understanding before the details.

The problem it solved

Earlier language models processed text sequentially — word one, then two, then three — carrying a running summary forward. That design has two fatal properties.

It cannot parallelize. Word 500 requires words 1 through 499 to be processed first. Training on enormous corpora becomes impossibly slow.

It forgets. Information from early in a long sequence has to survive hundreds of update steps to reach the end. In practice it degrades badly.

The transformer’s answer: process every position simultaneously, and let each position look directly at every other position. No sequential bottleneck, no information decay over distance. Word 500 accesses word 3 as easily as word 499.

That is the whole idea. Everything else is machinery serving it.

The path through the model

Follow a sequence from input to output.

1 · Tokens become vectors

Text is cut into tokens, each mapped to an embedding — a vector of several hundred to several thousand numbers. The sequence is now a grid: one row per token, one column per dimension.

2 · Position gets added

Because every position is processed simultaneously, nothing yet encodes order. the cat sat and sat the cat are currently identical sets of vectors.

So positional information is injected. See How Does a Model Know Word Order? — this is a genuinely non-obvious problem with several competing solutions.

3 · The layer stack

The same two-part block repeats dozens of times. This is the heart of the model.

Part A — attention. Every token examines every other token and pulls in what is relevant. The word it finds its referent. An adjective finds its noun. A closing bracket finds its opener. Details in Self-Attention, and the reason there are many parallel copies in Why Multiple Attention Heads?.

Part B — feedforward. Each token’s vector, now enriched by context, passes through a small network independently. Attention moves information between positions; the feedforward layer processes information within one. Most of the model’s parameters live here, and this is where much factual knowledge appears to be stored.

Both parts are wrapped with residual connections and normalization, which is what makes stacking dozens of layers trainable at all — see Layers, Residuals, and Depth.

Each layer refines the representation. Early layers resolve syntax and word sense; middle layers track entities and relationships; late layers assemble what is needed to predict the next token.

4 · Output

After the final layer, the last position’s vector is projected onto the vocabulary, producing one score per possible token. Softmax turns those into probabilities, a sampling step picks one, and the generation loop appends it and runs again.

Encoder, decoder, or both

The original paper described two stacks — an encoder reading input and a decoder producing output — because it targeted translation.

Language models kept only the decoder. A decoder-only model is causal: each position may attend to earlier positions and itself, never to later ones. That restriction is what makes next-token prediction coherent, since the model must not see the answer it is predicting.

Encoder-only models like BERT go the other way, letting every position see every other in both directions. Excellent for classification and embeddings, unsuitable for generation. When people say “an LLM,” they nearly always mean decoder-only.

What it costs

The design bought parallelism and long-range access, and paid for it in compute: every token attending to every token means work growing with the square of sequence length. See Why Long Contexts Cost So Much.

That single scaling property drives context window limits, pricing, and a large fraction of current research. KV caching is the main mitigation during generation.

What to remember

• Transformers process all positions at once and let each attend directly to any other — removing the sequential bottleneck and long-range decay of earlier designs.
• The path: tokens → embeddings → positional information → many (attention + feedforward) blocks → vocabulary probabilities.
• Attention moves information between positions; feedforward layers process it within one and hold most parameters.
• Language models are decoder-only and causal: each position sees only what precedes it.
• The cost is quadratic in sequence length.

Next: Self-Attention, Explained Visually — the mechanism at the center.