Recurrent Neural Networks

The first neural network built to read sequences one step at a time, carrying a hidden state forward. How RNNs work, what they made possible, and the flaw that limited them.

A plain neural network takes a fixed-size input and produces a fixed-size output. That is a problem for language, because sentences have no fixed length and word 20 can depend on word 2. You need an architecture that reads a sequence in order and remembers what it has seen. That was the recurrent neural network, and for years it was how machines processed text.

The core idea: a loop with memory

An RNN reads one token at a time. At each step it combines two things:

1. The current input — the vector for the word it is reading right now (often a word2vec-style embedding).
2. Its hidden state — a vector summarizing everything it has read so far.

It produces a new hidden state, which it carries into the next step. That is the “recurrence”: the output of one step feeds back as an input to the next.

Reading "the cat sat":

• Step 1: read the, combine with empty state → state h₁
• Step 2: read cat, combine with h₁ → state h₂ (now knows about “the cat”)
• Step 3: read sat, combine with h₂ → state h₃ (knows about “the cat sat”)

The same set of weights is used at every step. The network is not 3 layers deep for a 3-word sentence; it is one small network applied repeatedly, with the hidden state threading through. That weight sharing is what lets a single RNN handle sentences of any length.

Why the hidden state matters

The hidden state is the RNN’s entire memory of the past, compressed into one fixed-size vector. This is what let RNNs do things bag-of-words never could.

Consider predicting the last word of "I grew up in France, so I speak fluent ___". A bag-of-words model sees a pile of words with no order and cannot connect France to the blank. An RNN reads left to right, stores France in its hidden state, carries it forward, and uses it when it reaches the blank. Order and context, finally usable.

This unlocked a wave of applications: language modeling, sentiment classification, named-entity recognition, and — chained together — the sequence-to-sequence models that first made neural machine translation work.

Training through time

RNNs learn by backpropagation, but with a twist. To compute how a word 10 steps ago affected the current error, you have to trace the gradient back through all 10 steps of the recurrence. This is backpropagation through time: unroll the loop into a deep chain and propagate the error signal all the way back.

That chain is where the trouble starts.

The vanishing gradient problem

Here is the flaw that ultimately limited RNNs. When you propagate a gradient back through many steps, you multiply many numbers together at each step. Two bad things happen:

• If those numbers are consistently less than 1, the product shrinks toward zero. The gradient vanishes. The network cannot learn connections between distant words, because the signal linking them fades to nothing before it reaches back.
• If they are consistently greater than 1, the product blows up. The gradient explodes, and training becomes unstable.

Vanishing gradients are the more insidious problem. In practice a basic RNN could remember maybe 5 to 10 steps of context. Ask it to connect the subject at the start of a long paragraph to a verb at the end, and the link had already faded. The hidden state was a memory that leaked.

The two structural limits

Beyond vanishing gradients, RNNs had a second, quieter problem that mattered enormously later.

Long-range dependencies fade. The core weakness above — information from many steps ago is hard to preserve. LSTMs and GRUs were invented specifically to patch this, with gated memory that holds information longer.

Processing is inherently sequential. To compute the hidden state at step 100, you must first compute steps 1 through 99, in order. There is no way to parallelize across the sequence. On modern GPUs built for massively parallel math, this is crippling. You cannot spread one sentence across thousands of cores, because each step waits on the one before it.

That second limit is the one gated memory could never fix, and it is the deepest reason the field eventually moved to self-attention, which processes every position at once. Why Transformers Won traces that shift in full.

What to remember

• An RNN reads a sequence one token at a time, updating a hidden state that summarizes everything seen so far.
• The same weights are reused at every step, so one network handles any sequence length.
• The hidden state gave RNNs real memory of order and context, enabling translation, tagging, and language modeling.
• Vanishing gradients meant long-range connections faded — RNNs struggled to link distant words.
• Sequential processing meant they could not parallelize across a sequence, which GPUs punish severely. This limit outlived every fix.

Next: LSTM and GRU — gated memory cells built to hold information across far more steps.