How Similarity Search Actually Works

Cosine similarity, dot products, and why 'nearest' is a weaker signal in high dimensions than it sounds.

Retrieval rests on a single operation: given a query vector, score every stored vector for similarity, return the highest.

The scoring function is simple arithmetic. What it means, and where it misleads, is the part worth understanding.

Cosine similarity

Two vectors are compared by the angle between them rather than the distance between their endpoints.

Compute the dot product, then divide by both magnitudes. The result lands in −1 to 1:

• 1.0 — identical direction
• 0 — unrelated, perpendicular
• −1 — opposite direction

In practice, embedding similarities cluster in a narrow band. Unrelated text often scores 0.1–0.3 rather than 0, and related text 0.7–0.9. Genuinely negative scores are rare. This matters: absolute values are not interpretable across models. A score of 0.82 means nothing on its own. Only the ranking within one model’s space is meaningful, which is why fixed similarity thresholds are fragile and relative cutoffs work better.

Why angle rather than distance

Magnitude tends to encode something incidental. Word frequency, document length, and training artifacts all affect vector length without affecting meaning.

Two documents on the same topic, one three times longer, should count as similar. Euclidean distance would separate them; cosine treats them as near-identical in direction. Semantic content lives in direction, so direction is what gets measured.

Normalized vectors collapse the distinction entirely. Scale every vector to unit length at index time and cosine similarity becomes a plain dot product — cheaper, and Euclidean distance becomes a monotone function of it, so the two rank identically. Most systems normalize for exactly this reason.

Where “nearest” gets weak

High-dimensional geometry undermines the intuition, in ways that show up as real retrieval problems.

Distances compress. As dimension grows, the ratio between a point’s nearest and farthest neighbour shrinks toward 1. Everything is roughly the same distance from everything else. “Nearest” survives as a ranking but weakens as a distinction — the gap between rank 1 and rank 20 can be small enough to be noise.

This is the core argument for reranking: vector search is good at narrowing millions to fifty, and unreliable at picking the best five out of that fifty.

Near-orthogonality is the default. Two random directions in high dimensions are almost always nearly perpendicular. This is what gives the space capacity — many concepts coexist without interference — and it is why any measurable similarity is meaningful even when the number looks low.

Query and document phrasing differ. “How do I reset my password?” and “Password reset procedure: navigate to Settings” are related but structurally dissimilar. Questions and statements occupy somewhat different regions, and this asymmetry causes real misses. Some embedding models accept a prefix marking text as query versus document, trained specifically to close this gap — using it when available is free improvement.

Where vector search fails outright

Embeddings capture meaning, which means they systematically miss things that are not about meaning:

Exact identifiers. Error code E4021, function name parse_header, order number. The embedding of a rare identifier is nearly meaningless — it fragments into subword pieces with no semantic content.

Negation. “Documents that mention X” and “documents that do not mention X” embed very close together. Vector similarity has no logical operators.

Numeric and date constraints. “Reports after March 2026” is a filter, not a similarity question.

Rare proper nouns. Same problem as identifiers.

Every one of these is what keyword search does well. Which is why hybrid search — combining vector and keyword scores — outperforms either consistently. The standard combination method is reciprocal rank fusion: merge by each result’s rank in both lists rather than trying to reconcile incomparable scores. Simple, effective, and it avoids the score-normalization problem entirely.

Practical notes

Retrieve more than you need, then narrow. Fetch 30–50 candidates, rerank to the best 3–5. Recall first, precision second.

Avoid absolute thresholds. “Only include results above 0.8” breaks the moment you change embedding models. Use ranking, or a threshold relative to the top score.

Deduplicate. With chunk overlap, near-identical chunks both match and both consume context.

Test with real questions. The measurement that matters is whether the correct chunk appeared in the retrieved set for actual user questions. Everything else is proxy.

What to remember

• Cosine similarity measures angle, because magnitude encodes incidental properties while direction carries meaning.
• Scores are only comparable within one model’s space — absolute thresholds are fragile.
• High dimensions compress distances, so vector search narrows well but ranks the final few unreliably.
• It fails on exact identifiers, negation, and numeric constraints — exactly what keyword search handles.
• Hybrid search with rank fusion is the standard fix; retrieve broadly, then rerank.

Next: Why You Need a Reranker