Where Does the Vocabulary Come From?

Byte Pair Encoding in four steps, and why the resulting vocabulary is a permanent fossil of the text it was built from.

A model’s vocabulary is a fixed list — typically tens of thousands of entries — decided before training begins and never changed afterward. Nobody wrote it by hand. It was grown by an algorithm.

Understanding how it grew explains several things that otherwise look arbitrary: why some words cost more tokens than others, why non-English text is more expensive, and why a model’s tokenizer cannot be updated when language changes.

Byte Pair Encoding, completely

The dominant algorithm is BPE, and it fits in four steps:

1. Start with the smallest possible units — individual bytes.
2. Scan a large corpus. Count every adjacent pair of units. Find the most frequent.
3. Merge that pair into a single new unit. Record the merge. Add it to the vocabulary.
4. Repeat until the vocabulary reaches its target size.

That is the whole thing. Run it 50,000 times and you have a 50,000-entry vocabulary.

Watching it run

Take a tiny corpus containing low, lower, lowest, slow, slowest.

Starting units: l, o, w, e, r, s, t.

• The pair l + o is frequent → merge into lo
• Now lo + w is frequent → merge into low
• e + s appears in lowest, slowest → merge into es
• es + t → merge into est
• s + low → merge into slow

After five merges the vocabulary holds low, est, slow as single units. The word lowest now costs two tokens (low + est) instead of six characters. A word that never appeared, like glowing, still tokenizes fine — it just costs more pieces.

That is the property that makes BPE work: frequent things become cheap, rare things stay representable.

Why the vocabulary is a fossil

Step 2 counts frequencies in a specific corpus. Every merge decision is a fact about that text.

This has permanent consequences:

English got the most whole-word entries, because the corpora used were English-dominant. Common English words are single tokens. The same meaning in another language, especially a non-Latin script, fragments into more pieces — you pay more money and more context window for identical content.

Domain vocabulary depends on corpus composition. If a great deal of code was included, common programming identifiers became single tokens. Medical or legal terminology may or may not have earned entries depending on what was scraped.

The vocabulary cannot be updated. Every model parameter was trained against these exact token IDs. Adding an entry would mean the embedding table no longer matches. New words coined after the tokenizer was built will always fragment.

Quirks get frozen in. Tokenizers trained on raw web scrapes have picked up strange artifacts — usernames from scraped forums, fragments of markup — as single tokens. These sometimes produce genuinely odd model behavior when triggered, because such tokens appeared in the vocabulary but were rare in the training text that followed.

Variants worth knowing

Byte-level BPE starts from raw bytes rather than characters, guaranteeing any possible input is representable. No character is ever unknown, because worst case it decomposes into bytes. Most current models use this.

WordPiece is similar but chooses merges by which one most improves the training objective, rather than raw frequency. Used by BERT-family models.

SentencePiece treats the input as a raw stream without assuming spaces separate words, which matters a great deal for languages that do not use spaces — Japanese, Chinese, Thai.

Why vocabulary size is a tradeoff

Larger vocabulary means shorter sequences — good, because attention cost grows faster than linearly with length.

But larger vocabulary also means a bigger embedding table and a bigger output layer, since the model must produce a probability for every entry. Each additional entry costs parameters at both ends.

There is also a data problem: a rare entry appears in training too seldom for the model to learn a good representation of it. Beyond a point, extra entries are dead weight.

Current models land in the tens of thousands to low hundreds of thousands, with multilingual models pushing higher to give non-English scripts fairer representation.

What to remember

• BPE grows a vocabulary by repeatedly merging the most frequent adjacent pair.
• The result is a fossil of its training corpus — whatever was common there is cheap forever.
• It cannot be updated after training, because every parameter is tied to these exact token IDs.
• Vocabulary size trades sequence length against parameter count.

Next: Training vs Inference