The Geometry of Meaning

Embedding spaces have structure: directions carry relationships, clusters carry categories, and the geometry behaves in ways that are useful and occasionally misleading.

An embedding space is not a random scatter of points. It has structure, and that structure is what makes semantic search, RAG, and clustering work.

It also has properties that reliably mislead people. Both halves are worth knowing.

Directions mean things

The famous demonstration: king − man + woman lands near queen.

Read that geometrically. There is a direction in the space that corresponds roughly to “gender,” and it is approximately consistent — the vector from man to woman resembles the vector from king to queen, from actor to actress. Meaning is encoded not just in where points sit but in the offsets between them.

Similar structure appears for singular/plural, verb tense, comparative/superlative, and country/capital pairs. None of it was designed in. It emerged because encoding these regularities helps predict the next token.

The honest caveat: these analogies work far better in curated demonstrations than in general. Published results often exclude the input words from the answer set, which quietly does a lot of work. Directions are real and useful but noisier than the examples imply.

Clusters mean categories

Zoom out and points group. Animals near animals, programming languages near programming languages, emotions near emotions. Nobody labeled these regions; usage patterns produced them.

Hierarchy shows up too. Within an animal cluster, dog breeds cluster more tightly than the broader animal region. Categories nest, echoing how the words are actually used.

This is what makes clustering a practical application. Group support tickets or documents by embedding proximity and you get topical grouping without defining topics in advance.

Why cosine rather than distance

Similarity is nearly always measured by cosine — the angle between vectors — rather than straight-line distance.

The reason is that magnitude tends to encode something incidental. Common words often end up with different vector lengths than rare ones; longer documents can produce longer vectors. Neither is about meaning. Direction is where the semantic content lives.

Two documents on the same topic at very different lengths should count as similar, and cosine gives you that. See How Similarity Search Actually Works.

High-dimensional space is strange

Embeddings live in hundreds or thousands of dimensions, and intuitions built in three dimensions mislead badly.

Almost everything is nearly orthogonal. Pick two random directions in high dimensions and their cosine similarity is close to zero. This is what gives the space its capacity — enormous numbers of concepts can coexist without interfering, because there is room for them to be mutually unrelated.

Distances compress. As dimension grows, the ratio between the nearest and farthest neighbor of a point shrinks toward 1. “Nearest” becomes a weaker distinction than it sounds, which is part of why retrieval benefits from a second, more discriminating pass — see Why You Need a Reranker.

Volume lives at the surface. Almost all the volume of a high-dimensional ball sits near its boundary. Points are more spread out and less centrally clustered than a 3D mental picture suggests.

Superposition. Models appear to represent more features than they have dimensions, by encoding features in overlapping non-orthogonal directions and tolerating slight interference. This is an active research area and it complicates any simple “one direction per concept” story.

What 2D pictures do and do not show

Visualizations of embedding spaces are projections from hundreds of dimensions down to two, using techniques like t-SNE or UMAP.

They are genuinely useful for seeing cluster structure. They are also lossy in specific ways worth knowing: distances between clusters are often not meaningful, apparent cluster sizes are artifacts of the algorithm’s parameters, and running the same projection twice can produce visibly different layouts.

Read them as “these things group together,” never as “this thing is exactly this far from that thing.”

Bias is geometric

If the training text associated certain occupations with certain genders, that association exists as a measurable direction in the space. The same mechanism that gives you king − man + woman ≈ queen will give you stereotyped analogies for occupations.

This is not a separate defect layered on top. It is the same structure doing the same thing, applied to associations present in the source text. An embedding space is a compressed record of how people wrote, including the parts nobody would endorse.

Debiasing techniques exist — identifying and neutralizing specific directions — with partial success. Worth knowing they are partial.

What to remember

• Offsets between points encode relationships; clusters encode categories. Both emerged from training rather than being designed.
• Cosine similarity is preferred because magnitude often encodes incidental properties while direction carries meaning.
• High dimensions are unintuitive: near-orthogonality is the default, distances compress, and superposition lets models pack in more features than dimensions.
• 2D projections show grouping reliably and distances unreliably.
• Bias in the training text becomes measurable geometric structure.

Next: Prompt Basics — from internals to practice.