What Is an Embedding?

Embeddings turn meaning into coordinates, which is what lets a machine compute with language. The foundation under search, RAG, and attention.

A neural network cannot multiply the word dog. It can only do arithmetic on numbers.

Tokenization gets partway there by assigning dog an ID — say 5847. But that ID is a label, not a quantity. Token 5847 is not “more” than token 5846, and the two are not related just because their IDs are adjacent. Doing math on vocabulary IDs would be meaningless.

An embedding solves this. Instead of one arbitrary number per token, assign each token a list of hundreds of numbers — a vector — positioned so that distance corresponds to meaning.

Meaning as position

Picture a map where every word occupies a point, and the arrangement is not alphabetical but semantic.

dog and puppy land close together. dog and cat sit nearby — different animals, similar role in language. dog and democracy are far apart. Real embeddings use hundreds or thousands of dimensions rather than two, but the intuition survives the jump: nearby means similar.

Nobody assigns these positions by hand. They are learned. During training, tokens that appear in similar contexts get nudged toward each other, over and over, across enormous amounts of text. The arrangement is a residue of usage patterns.

This is the distributional hypothesis doing the work: words used in similar contexts tend to mean similar things. Turn that into an optimization objective and geometry falls out.

Directions carry meaning too

The surprising part is not just that similar things cluster. It is that directions in the space turn out to be meaningful.

The classic demonstration: take the vector for king, subtract man, add woman, and you land near queen. There appears to be a consistent “gender” direction, and a separate “royalty” direction, and they compose.

Similar structure shows up for singular/plural, present/past tense, and country/capital pairs. The model was never told these categories exist. They emerged because encoding them helps predict the next token.

Two caveats worth holding. These analogies are cleaner in demonstrations than in general — real embedding spaces are messier than the examples suggest. And the space inherits whatever associations were present in the training text, including harmful ones, because it is a compressed record of how people actually wrote.

Static vs contextual

Early embeddings gave each token exactly one vector. bank got one position, forever, whether you meant a riverbank or a savings account.

Modern models do something better. The initial embedding is a starting point, and then each layer of the network adjusts every token’s vector based on its neighbors. By the upper layers, bank in “river bank” and bank in “bank account” occupy genuinely different positions.

This adjustment mechanism is self-attention, which makes embeddings the direct prerequisite for understanding how transformers work. Attention is essentially a procedure for letting vectors update each other based on relevance.

So the pipeline runs: token → static embedding → contextually adjusted vector, layer after layer.

What embeddings are used for directly

Beyond being internal machinery, embeddings are a product in their own right.

Semantic search. Embed the query, embed every document, return the nearest neighbors. This finds relevant results even with no shared keywords — “how do I stop my code crashing” can match a document titled “exception handling,” because meaning drove the match, not string overlap.

Retrieval-augmented generation. RAG is built on exactly this: embed your documents, embed the incoming question, retrieve the closest chunks, and hand them to the model as context. Embeddings are the retrieval half of the system.

Clustering and deduplication. Group similar support tickets, find near-duplicate articles, detect topic drift over time.

Classification. Embed labeled examples, embed a new item, assign it the label of its nearest neighbors. Often startlingly effective for how little machinery it requires.

For all of these you need somewhere to keep the vectors and a fast way to find neighbors — see What Is a Vector Database? and How Similarity Search Actually Works.

Comparing vectors

Two vectors are compared by measuring how much they point in the same direction, rather than how far apart their endpoints are. That measure is cosine similarity: 1.0 means identical direction, 0 means unrelated, negative means opposed.

Direction is preferred over raw distance because vector magnitude often encodes something incidental — like how frequently a term appeared — while direction carries the semantic content. Two documents on the same topic at very different lengths should still count as similar.

What to remember

• Embeddings convert tokens into vectors positioned so that distance means similarity.
• Positions are learned from context, never hand-assigned; directions in the space encode relationships like gender or tense.
• Modern models adjust each vector by its context as it passes through layers — that adjustment step is attention.
• Semantic search, RAG, clustering, and classification are all direct applications.

Next: How Does an LLM Actually Write? — the generation loop these vectors feed.