State Space Models

An alternative to attention with linear cost and constant memory. Where it wins, where it loses, and why hybrids are winning.

Attention costs quadratic time and stores a growing cache. Both problems come from the same design decision: every token can look directly at every other token.

State space models take the opposite approach. They maintain a fixed-size hidden state and update it as tokens stream past. Cost per token is constant. Memory is constant. Total cost is linear in sequence length.

The core difference

An SSM processes a sequence recurrently: read a token, update the state, move on. The state is a compressed summary of everything seen.

That sounds like the recurrent networks transformers replaced, and the resemblance is real. Two things changed to make it work.

The state update is structured so it can be parallelized during training. Naive recurrence cannot be — step t needs step t−1, so training is sequential and slow. SSMs constrain the update to a form that allows computing all positions simultaneously via a scan operation. Training parallelizes; inference stays recurrent.

The update became input-dependent. Early SSMs used a fixed update rule regardless of content, which limited them badly. Mamba’s contribution was making the state transition depend on the current token — the model learns what to keep and what to discard based on what it is reading. This selectivity is what closed most of the quality gap with attention.

The tradeoff

Attention keeps everything and searches it. SSMs keep a summary and update it.

What SSMs win: constant memory during generation regardless of sequence length, so no KV cache and no growing per-request memory. Linear total cost. Very long sequences become tractable — audio, DNA, high-resolution signals where token counts run into the millions.

What they lose: the state is finite, so information must be discarded. Exact recall of a specific detail from far back is genuinely harder. Attention can retrieve token 3 at position 500,000 precisely because it never compressed it.

This is not a small caveat. Tasks requiring precise retrieval from long context — finding a specific fact in a document, copying an exact string — are where pure SSMs underperform, and those tasks are common in practice.

Hybrids

The result the field converged on: combine them. Mostly SSM layers for efficiency, with attention layers interspersed for precise recall.

The intuition is that most processing does not need exact long-range lookup, and the few layers that do can carry it. Hybrid models report attention-level quality at substantially better cost scaling, and several production models now use this shape.

This mirrors what happened with long-context attention, where the winning pattern was also hybrid — mostly cheap windowed layers, a few full-attention ones. Different mechanisms, same conclusion: pay for global reach only where it is needed.

Where SSMs matter most

Very long sequences. Genomics, audio, sensor streams, video. Where token counts make quadratic attention simply impossible.

Memory-constrained serving. No KV cache means far more concurrent requests per unit of memory — potentially a large operational advantage.

Edge and on-device. Constant memory is a strong property when memory is fixed and small.

Streaming. The recurrent formulation is naturally incremental, with no cache to manage.

What this means for you

For most application work, nothing yet. If you are calling a hosted API, the architecture is not your concern, and the dominant models remain attention-based or hybrid.

It matters if you self-host under memory pressure, work with extremely long sequences, or deploy on constrained hardware. And it is worth understanding as the clearest illustration of a real architectural tradeoff: remember everything and search it, or summarize and update. Attention chose the first, SSMs the second, and hybrids acknowledge that neither answer is right for every layer.

What to remember

• SSMs maintain a fixed-size state updated per token: constant memory, linear total cost, no KV cache.
• Two advances made them viable: a parallelizable training formulation, and input-dependent state updates (Mamba’s selectivity).
• The cost is compression — precise recall of distant details is genuinely weaker than attention’s.
• Hybrid SSM-plus-attention models are where the field landed, paying for exact recall only in a few layers.
• Most relevant for very long sequences, memory-constrained serving, and on-device deployment.

Next: Speculative Decoding