How Does a Model Know Word Order?
Attention is order-blind by construction. Position has to be injected separately, and how it is injected determines how far context can stretch.
On this page
Here is a fact about self-attention that is easy to miss and impossible to unsee: it has no concept of order.
Attention computes scores between pairs of tokens using dot products. Shuffle the input sequence and each pair produces exactly the same score. To the mechanism itself, dog bites man and man bites dog are the same bag of tokens.
That is a catastrophic property for language. So position must be supplied separately.
Why the obvious fix fails
The naive approach is to append the index — 1, 2, 3 — to each embedding.
It breaks for two reasons. Raw magnitudes grow without bound, so position 5000 would numerically dominate every semantic dimension. And the model has no reason to generalize: nothing about the number 5000 tells it that 5000 relates to 4999 the way 3 relates to 2.
What is actually needed is a representation where relative distance is recoverable, because that is what language cares about. “The adjective two words back” matters; “the adjective at absolute index 847” almost never does.
Sinusoidal encoding
The original transformer paper used a fixed pattern of sines and cosines at many different frequencies. Each position gets a unique vector, added directly to its embedding.
The construction has an appealing property: because of how sine and cosine compose, the encoding for position p+k is a linear transformation of the encoding for position p, with the transformation depending only on k. Relative offsets are therefore linearly recoverable — exactly the property that was needed.
It also requires no parameters and extends to any length. It worked, and it is where most explanations stop, but it is no longer what most models use.
Learned absolute positions
An alternative: treat each position as having its own learned embedding, trained like any other parameter.
Simple, and it performs well within the trained range. The fatal weakness is that it cannot extrapolate at all. A model trained to position 512 has no embedding for position 513 — the parameter does not exist. Context length becomes a hard architectural ceiling.
Rotary embeddings
Most current models use RoPE — rotary position embedding — and the idea is elegant.
Instead of adding a position vector, rotate each token’s query and key vectors by an angle proportional to their position. Take pairs of dimensions, treat them as a 2D plane, and rotate by an amount determined by the position index.
The payoff falls out of trigonometry. When a query at position m meets a key at position n, their dot product depends only on m − n — the relative distance. Absolute positions cancel. Position information is baked into the attention computation itself rather than added to the input, and relative distance is what the mechanism naturally sees.
This turns out to matter enormously for long context. Because RoPE encodes relative rather than absolute position, its behavior can be stretched. Scaling the rotation frequencies lets a model trained at one context length operate at a longer one with modest additional tuning — which is how many models shipped large context windows without retraining from scratch.
Two other approaches worth naming. ALiBi skips positional vectors entirely and instead applies a distance-proportional penalty to attention scores, biasing every head toward nearby tokens; it extrapolates well and is very cheap. Relative position embeddings add learned biases per relative offset, effective but heavier.
Why this determines context length
Positional encoding is one of the two real constraints on how long a context can be. The other is quadratic attention cost.
Learned absolute encoding sets a hard wall. Sinusoidal and RoPE have no wall in principle, but a model trained mostly on short sequences still degrades on long ones — the positional patterns at distance 100,000 were simply never practiced.
This is the mechanism behind a familiar complaint: a model advertising a very large context window that seems to attend poorly to the middle of it. The positions exist and are representable; whether the model learned to use them well at that distance is a separate question, and it depends on training data, not architecture.
What to remember
- Attention is order-blind by construction; position must be injected separately.
- Naive indices fail because magnitudes grow and relative distance is not recoverable.
- Sinusoidal encoding makes relative offsets linearly recoverable and needs no parameters.
- Learned absolute encoding cannot extrapolate past its trained length — a hard ceiling.
- RoPE rotates queries and keys so attention scores depend only on relative distance, which is what makes context extension practical.
- Long advertised context does not guarantee good long-range use; that depends on training.