What Is a KV Cache?
Generating token 500 should not require recomputing tokens 1 through 499. The cache that prevents it, and the memory it costs instead.
On this page
Generation is a loop: predict a token, append it, run the model again on the longer sequence.
Taken literally, that is enormously wasteful. Producing token 500 would mean recomputing everything for tokens 1 through 499 — work already done 499 times.
The KV cache eliminates it, and understanding why it works explains most of what makes long-context inference expensive.
The redundancy
In self-attention, every token produces a query, a key, and a value from its own vector.
Causal masking means each token attends only to itself and earlier positions. Token 3 can never see token 4. So token 3’s key and value do not depend on anything that comes after it — they are the same whether the sequence is 10 tokens long or 10,000.
Therefore: once computed, a token’s keys and values never change.
That is the entire insight. Store them.
Two phases
With caching, generation splits into two operations with very different performance characteristics.
Prefill. Process the whole input prompt at once. Compute keys and values for every position and store them. This is highly parallel — all positions at the same time — and therefore fast per token. It is also the phase whose cost scales with the square of prompt length, since every token attends to every other.
Decode. For each new token: compute only its query, key, and value. Attend against the cached keys and values for all previous positions. Append the new pair to the cache. One token’s worth of work instead of the whole sequence.
This is why a long prompt returns its first token slowly and then streams quickly afterward. Prefill is a single large batch of work; decode is a fast per-token loop. Providers price and report these separately — time-to-first-token measures prefill, tokens-per-second measures decode.
What it costs
The cache trades computation for memory, and the memory is substantial.
Size scales with the product of: sequence length, number of layers, number of attention heads, head dimension, two (keys and values), and bytes per number.
Every one of those multiplies. A long context on a large model can require a KV cache larger than the model’s own parameters. For serving, this is usually the binding constraint — not model size, but cache size times the number of concurrent users.
Two consequences follow directly. Long contexts are memory-expensive at inference even after prefill is paid for. And serving many users simultaneously means many caches held at once, which is why providers impose context limits well below what the architecture could theoretically handle.
How the pressure gets relieved
The cache being the bottleneck drove real architectural change.
Grouped-query attention has heads share keys and values in small groups rather than each maintaining its own. Cache size drops by the group factor with modest quality cost. Multi-query attention is the extreme: one shared set of keys and values for all heads. Most current large models use one of these.
Paged attention borrows from virtual memory, allocating cache in fixed blocks rather than one contiguous reservation per request. This eliminates the waste from over-provisioning for a maximum length that most requests never reach, and it is what allows serving systems to pack many more concurrent requests onto the same hardware.
Quantized caches store keys and values at lower precision, trading some accuracy for capacity.
Prefix caching is the one most visible to you as a user. If many requests share an identical prefix — the same system prompt, the same retrieved documents — its keys and values can be computed once and reused. This is the mechanism behind prompt caching features that reduce cost on repeated prefixes, and it is why stable content belongs at the start of your prompt and variable content at the end. See Cutting Your API Bill.
Why conversations still get expensive
The cache removes redundant computation within a single generation. It does not remove the cost of re-sending conversation history.
Each turn of a chat resends the full transcript. Without prefix caching, that means paying prefill on the entire history again. With prefix caching, the shared prefix is cheap but the growing tail is not. Either way, cost climbs as the conversation lengthens — the cache changes the constant, not the shape.
What to remember
- Past tokens’ keys and values never change, because causal masking prevents them from depending on later tokens — so they can be cached.
- Prefill processes the prompt in parallel; decode generates one token at a time against the cache.
- Cache size scales with length × layers × heads × head dimension, and often exceeds the model’s own parameter memory.
- Grouped-query attention, paged attention, and prefix caching all exist to relieve this bottleneck.
- Putting stable content first in a prompt makes prefix caching effective.