Sequence to Sequence and Encoder-Decoder

How two recurrent networks were chained to turn one sequence into another — the architecture behind neural machine translation, and the bottleneck that led to attention.

Classifying a sentence gives you one label. But translation, summarization, and question answering need something harder: take one sequence in, produce a different sequence out, of a different length, in a different language. The sequence-to-sequence architecture, introduced in 2014, was the design that made this work with neural networks, and it introduced the encoder-decoder split that survives in models today.

Two networks, one handoff

The core idea is to use two RNNs (in practice LSTMs) with a division of labor.

The encoder reads the entire input sequence, one token at a time, updating its hidden state as it goes. When it reaches the end, its final hidden state is a single fixed-size vector meant to summarize the whole input. This vector is called the context vector, or thought vector.

The decoder is a second RNN, initialized with that context vector. It generates the output sequence one token at a time, feeding each word it produces back in as the input for the next step, until it emits a special end-of-sequence token.

For translating "the cat sat" into French:

• The encoder reads the, cat, sat, compressing all three into one context vector.
• The decoder takes that vector and produces le, then chat, then s'est, then assis, then stop.

The two networks are trained together end to end. The encoder learns to pack meaning into the context vector; the decoder learns to unpack it into a fluent output.

Why the encoder-decoder split matters

Separating “understand the input” from “produce the output” is a genuinely powerful factorization, and it is why the pattern outlived the RNNs it was born with.

It handles variable and mismatched lengths naturally. A 5-word English sentence can become an 8-word French one; the decoder just keeps going until it decides to stop. Bag-of-words and plain classifiers could never do this.

It decouples the two languages. The encoder only has to understand the source; the decoder only has to generate the target. Swap the decoder and you retarget to a new output language while reusing the same understanding.

This separation reappears everywhere later. Encoder-decoder architectures in the Transformer era keep exactly this structure for translation. Even decoder-only models like GPT are, in a sense, the decoder half taken on its own.

The bottleneck

Look again at the handoff. Everything the encoder understood about the input must pass through one fixed-size context vector. The whole sentence — every noun, every clause, every dependency — gets squeezed into, say, 512 numbers.

For a short sentence this is fine. For a long one it is a disaster. A 40-word sentence has to compress into the same vector as a 4-word one, and detail gets crushed. Researchers observed exactly this: translation quality was decent for short inputs and fell off sharply as sentences grew longer. The single context vector was a funnel that everything had to pass through, and it was too narrow.

There was also an obvious waste. When the decoder generates the French word for “cat,” it would benefit from looking directly at the encoder’s representation of “cat.” But it has no access to the encoder’s intermediate states — only the final squeezed summary. All that per-word information the encoder computed gets thrown away after the last step.

The fix that changed everything

The solution, proposed in 2014, was to stop forcing everything through one vector. Instead of giving the decoder only the final context vector, let it look back at all of the encoder’s hidden states, and at each output step, decide which input words to focus on.

Generating “chat”? Focus on the encoder’s state for “cat.” Generating a verb? Focus on the verb. The decoder computes a fresh, weighted blend of the encoder states for every word it produces.

That mechanism is attention, and it removed the bottleneck by giving the decoder direct, selective access to the entire input. It is the single most consequential idea in this whole layer, and it gets its own treatment in The Origin of Attention.

What to remember

• Seq2seq chains an encoder RNN that reads the input into a context vector and a decoder RNN that generates the output from it.
• It handles variable, mismatched input and output lengths, which plain classifiers cannot.
• The encoder-decoder split — understand, then generate — is a durable pattern that outlived RNNs.
• The flaw: the entire input is squeezed through one fixed-size context vector, which crushes long sentences.
• The fix was attention: let the decoder look back at all encoder states and focus on the relevant ones per output word.

Next: The Origin of Attention — how letting the decoder look back removed the bottleneck and set the stage for the Transformer.