Why You Need a Reranker

Vector search narrows millions to dozens quickly and ranks the final few badly. A second pass that reads query and document together fixes it.

Adding a reranker is usually the single largest quality improvement available to a working RAG system, per unit of effort.

The reason lies in a structural limitation of how vector search computes similarity.

The bi-encoder problem

In vector search, documents are embedded independently, ahead of time. Each chunk becomes a vector without any knowledge of what will eventually be asked. At query time, the question is embedded separately and compared.

This is a bi-encoder: two separate encoding passes that never meet. It is what makes the approach fast — document vectors are precomputed once and reused for every query — and it is also its ceiling.

Because each document was compressed into a fixed vector before the question existed, that vector has to serve every possible question equally. Nuance specific to this query was averaged away at index time. The comparison is between two summaries, never between the actual texts.

Add the high-dimensional distance compression problem — where nearest and farthest neighbours differ less than intuition suggests — and the result is consistent: vector search is excellent at narrowing millions to fifty, and unreliable at choosing the best five from that fifty.

What a cross-encoder does differently

A reranker takes the query and one document together, as a single input, and outputs a relevance score.

Because both texts are present in the same forward pass, attention can operate across them. The model can register that this document’s third paragraph directly answers the question’s specific condition, that a term appears in a different sense than the question intends, that the document is about the right topic but the wrong version.

No compression happened first. Nothing was averaged away. This is why cross-encoders rank substantially better.

The cost is that nothing can be precomputed. Every query-document pair needs its own forward pass. Scoring a million documents this way is impossible — which is exactly why the two stages are complementary rather than competing.

The two-stage pattern

1M documents
  → vector search (fast, approximate)     → top 50
  → cross-encoder rerank (slow, accurate) → top 5
  → send to model

Retrieval optimizes recall: get the right document into the candidate set. Reranking optimizes precision: order that set correctly.

Practical shape: retrieve 30–100 candidates, rerank down to 3–10. Reranking 50 pairs is a modest, bounded cost — a fraction of the generation cost that follows.

Why it matters more than it sounds

Two mechanical reasons the final ordering carries real weight.

Retrieved chunks fill the context window, and attention is unevenly distributed across a long context. Material at the beginning and end is used more reliably than material in the middle. So the order of chunks affects what the model actually uses, not just what it receives.

And irrelevant chunks are not neutral. They consume budget, dilute attention, and occasionally get used — producing a confidently wrong answer sourced from a passage that merely looked related. Fewer, better chunks frequently beat more chunks.

Options

Hosted reranking APIs. Several providers offer purpose-built reranking endpoints. Simplest path, priced per document scored.

Open cross-encoder models. Small models trained for relevance scoring, runnable locally. Fast enough on modest hardware for typical candidate-set sizes, and no per-query cost.

LLM-as-reranker. Ask a general model to score or order the candidates. Flexible, works with no extra infrastructure, and more expensive and slower than a purpose-built reranker. Reasonable for low volume or for prototyping.

When to skip it

Very small corpora. With a few hundred chunks, retrieval is already picking from a narrow set.

Hard latency budgets. Reranking adds a round trip. If you are fighting for every 50ms, it may not fit.

Retrieval is not the bottleneck. Diagnose first. If the correct chunk is not in the retrieved candidates at all, reranking cannot help — that is a chunking or hybrid search problem. Reranking only reorders what retrieval found.

That diagnostic order matters and is often skipped: check whether the right chunk was retrieved, then improve ranking.

What to remember

• Vector search embeds documents before knowing the question, so query-specific nuance is averaged away.
• A cross-encoder processes query and document together, letting attention work across both — much better ranking, no precomputation possible.
• Standard pattern: retrieve 30–100 for recall, rerank to 3–10 for precision.
• Order matters because attention is uneven across long contexts, and irrelevant chunks actively hurt.
• Reranking cannot rescue a chunk that retrieval never surfaced — verify retrieval first.

Next: RAG or Fine-Tuning?