The Origin of Attention

Before self-attention, there was Bahdanau attention: a 2014 fix for translation that let the decoder look back at the whole input. Where the idea came from and how it worked.

Attention is famous today as the engine of the Transformer, but it was invented three years earlier to solve a specific, mundane problem: sequence-to-sequence translation broke down on long sentences. The fix, published by Bahdanau and colleagues in 2014, is the direct ancestor of the self-attention you know. Seeing it in its original form makes the modern version far clearer.

The problem it solved

Recall the seq2seq bottleneck. An encoder RNN reads the whole input and crushes it into one fixed-size context vector, which the decoder then unpacks. For a long sentence, that single vector cannot hold everything, and translation quality collapsed as sentences grew.

The waste was glaring. The encoder computed a hidden state at every input word, rich with per-word detail, then threw all of them away and kept only the last one. When the decoder was generating the French word for “cat,” the encoder’s state right at “cat” would have been exactly what it needed — but that state was gone.

The idea: let the decoder look back

Bahdanau’s move was to keep every encoder hidden state and give the decoder access to all of them. At each step of generating the output, the decoder does not rely on one frozen summary. It builds a fresh context vector, custom-made for the word it is about to produce, by blending the encoder states it currently cares about.

The recipe, at each decoder step:

1. Score. Compare the decoder’s current state against every encoder hidden state, producing a relevance score for each input word. Bahdanau computed this score with a small neural network — a learned function that takes the two states and outputs how well they match.
2. Normalize. Pass the scores through a softmax so they become weights that sum to 1. These are the attention weights: how much to focus on each input word right now.
3. Blend. Take a weighted sum of the encoder states using those weights. That is the context vector for this step — mostly the states that scored high.
4. Generate. The decoder uses this custom context vector to produce the next word.

Every output word gets its own context vector. Generating “chat” pulls heavily from the encoder state at “cat”; generating a verb pulls from the verb. The bottleneck is gone, because the decoder now reaches directly into the full input and selects what it needs.

Alignment falls out for free

A striking bonus: the attention weights are interpretable. Plot them as a matrix of output words against input words, and you see the model’s alignment — which source words it looked at to produce each target word.

For translation this recovers something linguists used to build by hand. Translating English to French, the attention matrix lights up roughly along the diagonal, with off-diagonal jumps exactly where the two languages reorder words. The model was never told how the languages align; it learned to look in the right places, and the weights reveal it. This was early, welcome evidence that attention was doing something sensible.

Additive attention vs the scaled dot product

Bahdanau’s scoring function is called additive (or “concat”) attention: it feeds the decoder state and an encoder state into a small feed-forward network that learns to score their match. It works well but adds parameters and computation for every comparison.

Compare this to the self-attention in modern Transformers, which uses scaled dot-product attention. There, the score is just the dot product of a query and a key vector, divided by the square root of the dimension. No extra network — a bare dot product, which is a cheap, highly parallelizable matrix multiply.

The differences are worth holding side by side:

• Scoring. Additive uses a learned mini-network; scaled dot-product uses a plain dot product plus a scaling factor.
• Cost. Dot products are massively faster on GPUs, which is part of why the later version scaled.
• Direction. Bahdanau attention is cross-attention: the decoder (one sequence) attends to the encoder (another sequence). Self-attention has a sequence attend to itself.

That last point is the conceptual leap the Transformer made. Bahdanau attention connected two sequences. Self-attention applied the same looking-back trick within a single sequence, letting every word attend to every other word — and then dropped the RNN entirely.

The bridge to the Transformer

Bahdanau attention was bolted on top of RNNs; it still read the input sequentially, one step at a time, and just added a smarter handoff. It relieved the bottleneck without removing the RNN’s deeper flaw: sequential, unparallelizable processing.

In 2017, “Attention Is All You Need” asked the obvious next question. If attention is what makes translation work, do we need the RNN at all? Remove it, keep only attention, and you get a model that processes every position in parallel. That is the Transformer, and the story of why it won is in Why Transformers Won.

What to remember

• Attention was invented in 2014 to fix the seq2seq bottleneck, not for Transformers.
• The idea: keep all encoder states and let the decoder build a fresh weighted context vector for each output word.
• The steps — score, softmax, blend — are the same shape as modern attention.
• Bahdanau used additive scoring (a small network) and cross-attention (decoder attends to encoder); Transformers use scaled dot-product scoring and self-attention.
• The attention weights double as an interpretable alignment between input and output.

Next: Why Transformers Won — what happened when researchers kept attention and threw the RNN away.