Why Multiple Attention Heads?

One attention pass produces one blend. Running many in parallel lets a model track grammar, reference, and position at the same time.

A single self-attention pass produces one set of weights per token — one answer to “what is relevant to me.”

That is a real limitation. Consider:

The keys to the cabinet were on the table, and she grabbed them quickly.

The word them needs several relationships at once: which noun it refers to (keys, not cabinet), what verb governs it (grabbed), grammatical number agreement, and the physical setting. One weighted blend has to compress all of that into a single distribution. Attend strongly to keys and you dilute everything else.

Many heads, in parallel

The solution is to run attention many times over, simultaneously, with independent learned projections.

Each head gets its own query, key, and value matrices. Each therefore computes its own scores, its own weights, its own blend. A model with 32 heads performs 32 independent attention operations per layer.

The outputs are concatenated and passed through one more projection that mixes them back into a single vector per token.

Cost stays roughly flat, because each head works in a smaller subspace. If the model dimension is 4096 and there are 32 heads, each head operates in 128 dimensions. Total work is comparable to one full-width attention pass, but split into 32 specialized views instead of one averaged one.

That is the trade being made: the same compute buys many narrow perspectives rather than one wide one.

What heads specialize in

Nobody assigns roles. Specialization emerges during training because differentiated heads reduce prediction error. When researchers inspect trained models, recognizable patterns recur:

• Positional heads attend to the immediately preceding token, or a fixed offset back.
• Syntactic heads connect verbs to subjects, adjectives to nouns, prepositions to objects.
• Coreference heads link pronouns to their referents.
• Delimiter heads match brackets, quotes, and other paired structures.
• Rare-token heads attend to unusual tokens, which are often the most informative ones present.
• Induction heads find earlier occurrences of the current pattern and predict what followed last time — a mechanism closely tied to in-context learning and thus to why few-shot prompting works.

Two honest caveats. Many heads have no clean interpretation and appear to compute something distributed. And heads are frequently redundant — a good number can be pruned from a trained model with modest quality loss, which suggests training produces more than strictly needed.

Heads across layers

Specialization is not uniform through depth.

Early layers lean local: adjacent tokens, word sense disambiguation, basic syntax. Middle layers do the relational work — entity tracking, longer dependencies, coreference. Late layers concentrate on assembling whatever is needed for the immediate next-token prediction.

Roughly: early layers determine what each word is, middle layers determine how words relate, late layers determine what comes next.

The memory consequence

Every head maintains its own keys and values, and during generation all of them get cached. The KV cache therefore scales with head count, layer count, and sequence length together — which makes it the dominant memory consumer for long contexts.

This pressure produced architectural responses. Multi-query attention shares one set of keys and values across all heads while keeping separate queries, cutting cache size substantially. Grouped-query attention is the middle ground — heads share keys and values in small groups. Most recent large models use one of these rather than fully independent per-head keys and values, trading a little expressiveness for a large memory saving.

What to remember

• One attention pass yields one blend; multiple heads let a token track several relationships simultaneously.
• Each head works in a smaller subspace, so many heads cost about the same as one wide pass.
• Specialization is emergent, not assigned: positional, syntactic, coreference, delimiter, and induction heads recur across models.
• Early layers handle local structure, middle layers relationships, late layers next-token assembly.
• Head count multiplies KV cache size, which is why grouped-query and multi-query attention exist.

Next: How Does a Model Know Word Order?