Why Long Contexts Cost So Much

Attention compares every token with every token, so doubling the context quadruples the work. The constraint behind every context limit and price tier.

Self-attention has every token compare itself against every token. For n tokens that is n² comparisons.

That single fact sets context window limits, shapes API pricing, and drives a large fraction of current architecture research.

The arithmetic

Double the sequence and attention work quadruples.

A 100× longer input is not 100× more attention work. It is 10,000×.

The scores also have to be stored, not just computed — an n×n matrix per head per layer. At long context that matrix alone dwarfs everything else in memory, which is why naive implementations run out of memory before they run out of time.

Where the quadratic actually bites

An important qualification: attention is not the only cost, and at short lengths it is not the dominant one.

The feedforward layers scale linearly with sequence length, and they hold most of the parameters. For a few hundred tokens, linear feedforward work dominates and the quadratic term is negligible.

The crossover arrives in the low thousands of tokens. Past that, attention takes over and grows without bound relative to everything else. So “attention is the bottleneck” is true for long contexts specifically, not for all inference.

It also bites unevenly across the two phases. During prefill, the whole prompt is processed at once and the full n² cost lands immediately — which is why time-to-first-token climbs steeply with prompt length. During decode, each new token attends against n cached positions, so per-token cost grows linearly while the cumulative cost over a full generation is again quadratic.

What made long context practical

Two categories of solution, and the distinction matters.

Exact, but better engineered

FlashAttention is the important one. It computes mathematically identical attention while never materializing the full n×n matrix in slow memory. Instead it processes attention in tiles that fit in fast on-chip memory, computing partial softmax results and combining them.

The computation is still quadratic. But memory movement — which is what actually limits GPU throughput in practice — drops dramatically. This is not an approximation; the output is exact. A large share of the jump from thousand-token to hundred-thousand-token contexts came from this class of optimization rather than from any architectural change.

Approximate, by attending less

These change what attention computes:

Sliding window — each token attends only to the nearest k positions. Linear cost, but distant information is only reachable indirectly through stacked layers.

Sparse patterns — attend to a fixed subset: nearby tokens plus a few global anchors.

Global tokens — a small number of positions that everything attends to, acting as an information bottleneck through which distant context must pass.

Linear attention — reformulate to avoid the pairwise matrix entirely, achieving linear cost at the price of some expressiveness.

Several production models mix these, alternating full-attention layers with windowed ones so most layers are cheap while some retain global reach.

Why quality can lag the advertised limit

A model may accept 200,000 tokens and still use the middle of them poorly.

Two independent reasons. Approximate attention mechanisms genuinely do not look everywhere. And even with exact attention, the model must have learned to use distant positions — positional encodings at extreme distance may have been rare in training. The capacity exists; the practiced skill may not.

This is the source of a common frustration: information buried mid-context being ignored. Practical response is to place the most important material near the beginning or end of the prompt rather than trusting uniform attention across a very long input.

What this means for you

Prompt length is a real cost, not a rounding error. Padding a prompt with marginally relevant context degrades both cost and quality.

Retrieval beats stuffing. Selecting the right 2,000 tokens with RAG usually outperforms sending 100,000 and hoping. Cheaper and more accurate at once.

Long-context pricing is not arbitrary. Providers frequently charge more above a length threshold because the underlying cost genuinely changes shape there.

What to remember

• Attention cost grows with the square of sequence length; 10× longer is 100× the attention work.
• Feedforward layers dominate at short lengths — the quadratic term takes over in the low thousands of tokens.
• FlashAttention made long context practical by cutting memory movement without approximating.
• Sliding-window and sparse attention trade completeness for linear cost.
• Accepting a long context does not guarantee using it well; put important content at the edges.

Next: The Geometry of Meaning