How Long Context Became Possible

Context windows grew from thousands to millions of tokens. Four changes made that happen, and none of them repealed the quadratic.

Early transformers handled a few thousand tokens. Current models accept hundreds of thousands, some more than a million.

Attention cost is still quadratic. Nothing repealed that. Four separate advances made long context practical anyway, and they attack different constraints.

1 · Memory-efficient exact attention

The most important, and the least glamorous.

FlashAttention computes mathematically identical attention without ever materializing the full n×n score matrix in slow memory. It processes attention in tiles sized to fit in fast on-chip memory, computing partial softmax results and combining them correctly.

The computation remains quadratic. Memory movement — the actual bottleneck on modern accelerators — drops dramatically, and memory usage becomes linear rather than quadratic in sequence length.

This is not an approximation. Output is exact. A large share of the jump from thousands to hundreds of thousands of tokens came from this class of work rather than from any architectural change, which is why it belongs first.

2 · Positional encodings that extend

A model trained at 4,000 tokens has no idea what position 100,000 means, unless its positional scheme generalizes.

RoPE encodes relative rather than absolute position, which makes it stretchable: scaling the rotation frequencies lets a model trained at one length operate at a longer one with modest additional tuning. This is how many models shipped large context windows without retraining from scratch.

Learned absolute positional embeddings cannot do this at all — position 4,001 has no parameter. That limitation is why they fell out of use.

3 · KV cache reduction

At long context, the KV cache becomes the binding memory constraint — often exceeding the model’s own weights.

Grouped-query attention has heads share keys and values in small groups rather than each maintaining its own set, cutting cache size by the group factor for modest quality cost. Multi-query attention is the extreme case with one shared set. Most current large models use one of these.

Paged attention allocates cache in fixed blocks rather than reserving a contiguous maximum per request, eliminating the waste of provisioning for lengths most requests never reach. This is what lets serving systems pack many concurrent long-context requests onto the same hardware.

Cache quantization stores keys and values at reduced precision.

4 · Attention that skips

The approximate approaches, which change what attention computes rather than how efficiently it computes it.

Sliding window attention restricts each token to nearby positions — linear cost, with distant information reachable only indirectly through stacked layers. Global tokens designate a few positions everything attends to, forming a bottleneck for long-range information.

The practical pattern in production models is hybrid: most layers use cheap windowed attention, a few use full attention. Cost drops substantially while some layers retain global reach.

Why quality lags the number

A model accepting a million tokens may use the middle of them poorly, for two independent reasons.

Approximate attention genuinely does not look everywhere. And even with exact attention, the model must have learned to use distant positions — positional patterns at extreme distance were rare in training data. The capacity exists; the practiced skill may not.

This produces the well-documented pattern of information at the beginning and end of a long context being used more reliably than material buried in the middle.

Advertised context length is a capacity claim, not a quality claim. They are separate measurements, and only the first appears in marketing.

Practical consequences

Retrieval still beats stuffing. Selecting the right few thousand tokens with RAG usually outperforms sending a million and hoping — cheaper, faster, and more accurate.

Prefill dominates long-context latency. Time to first token climbs steeply with prompt length because the full quadratic cost lands at once. Decode speed is barely affected.

Long-context pricing reflects real cost. Providers frequently charge more above a threshold because the cost genuinely changes shape there.

Place important content at the edges. A practical accommodation of how attention actually distributes.

What to remember

• Attention is still quadratic; four separate advances made long context workable.
• FlashAttention cut memory movement without approximating — the largest single contributor.
• RoPE extends because it encodes relative position; grouped-query and paged attention relieved the KV cache bottleneck.
• Hybrid full-plus-windowed attention is the common production pattern.
• Accepting long input does not mean using it well; put critical material at the edges and prefer retrieval to stuffing.

Next: State Space Models