LSTM and GRU: Memory That Lasts

Recurrent networks that forgot too fast got a fix: gated memory cells. How LSTMs and GRUs decide what to keep, what to discard, and why they ruled NLP for a decade.

A basic RNN has a leaky memory. Its hidden state gets overwritten a little at every step, so information from far back fades before it can be used, and the vanishing gradient problem means the network cannot even learn to hold onto it. The LSTM, introduced in 1997 and dominant by the mid-2010s, was the fix: give the network an explicit memory it can choose to protect.

The problem, stated precisely

In a plain RNN, the hidden state is rewritten wholesale at each step. There is no way to say “keep this one fact untouched for the next 40 words.” Every step smears the past a little more. The network wants to carry France forward to predict French, but by the time it gets there, France has been diluted beyond recognition.

The LSTM’s answer is a separate cell state — a memory conveyor belt that runs straight through the sequence with only minor, deliberate edits. Information can ride it for many steps nearly unchanged, which is exactly what long-range dependencies need.

Gates: learned valves on the memory

An LSTM controls its cell state with three gates. Each gate is a small learned layer that outputs numbers between 0 and 1 — think of them as valves, where 0 means “block completely” and 1 means “let through fully.” The gates are computed from the current input and previous hidden state, so the network learns when to open and close them.

• Forget gate. Looks at the new input and decides what to erase from the cell state. Reading a new subject in a sentence, it might flush the old subject’s gender.
• Input gate. Decides what new information to write into the cell state. The new subject’s gender gets stored.
• Output gate. Decides what part of the cell state to expose as the hidden state for this step’s prediction.

The key trick is the forget gate’s path. Because the cell state is mostly added to rather than repeatedly multiplied, gradients can flow backward across many steps without vanishing. The gates learn to keep that path open for information that matters. This is the mechanism, not a metaphor: the additive cell state is what defeats the vanishing gradient.

A concrete pass

Take "The keys, which were on the table, are ___". To choose are over is, the network must remember the subject is plural (keys) across the whole intervening clause.

• At keys: the input gate writes “plural subject” into the cell state.
• Through which were on the table: the forget gate keeps the plural fact, letting the clause’s words pass without overwriting it.
• At the blank: the output gate surfaces “plural subject,” and the model predicts are.

A plain RNN would have lost the plural marker somewhere in the middle clause. The LSTM protected it on the cell state.

GRU: the streamlined cousin

The GRU (gated recurrent unit), from 2014, simplifies the LSTM. It merges the cell state and hidden state into one, and uses two gates instead of three:

• Reset gate — how much past state to ignore when computing the new candidate.
• Update gate — how much to blend old state versus new, doing the job of the LSTM’s forget and input gates at once.

Fewer gates means fewer parameters and faster training. In practice GRUs and LSTMs perform comparably on most tasks; GRUs are often preferred when data or compute is limited, LSTMs when the sequences are long and complex. The choice is usually empirical, not principled.

What they made possible

Gated RNNs were the backbone of applied NLP for roughly a decade. They powered production machine translation, speech recognition, and text generation. Stacked into encoder-decoder pairs, they became the sequence-to-sequence models that Google Translate ran on. If you used a language app between roughly 2015 and 2018, an LSTM was probably involved.

The limit they could not cross

LSTMs fixed memory. They did not fix sequentiality.

To compute step 100, an LSTM still has to compute steps 1 through 99 first, in order, because each cell state depends on the previous one. This is inherent to recurrence, and no gating scheme removes it. On GPUs designed to do thousands of operations at once, this forced serialization is a hard ceiling on training speed and sequence length.

There is also a subtler limit. Even with gates, squeezing an entire long sequence into one fixed-size cell state is a bottleneck. The further back a detail is, the more it competes with everything since for the same limited memory. Gating stretched the useful range from ~10 steps to a few hundred, but it did not make distant access free the way self-attention later would, where any position reaches any other in a single step.

Both limits — sequential computation and the fixed-size memory bottleneck — are what the Transformer was built to escape. Why Transformers Won puts the whole comparison side by side.

What to remember

• LSTMs add an explicit cell state, a memory that runs through the sequence with only deliberate edits.
• Three gates (forget, input, output) learn what to erase, write, and expose; the additive cell state defeats vanishing gradients.
• GRUs do the same job with two gates and fewer parameters, performing comparably in practice.
• Gated RNNs ran production translation and speech for about a decade.
• They fixed memory but not sequential processing, and the fixed-size state is still a bottleneck. Those limits ended the RNN era.

Next: Sequence to Sequence — chaining two RNNs to turn one sequence into another, the architecture behind neural translation.