Query Rewriting and Expansion

Users ask badly. Rewriting the query before retrieval fixes more failures than tuning the retriever.

Retrieval assumes the query is a good search key. Real queries frequently are not.

They are too short. They use pronouns referring to earlier turns. They ask two things at once. They are phrased as questions while documents are phrased as statements — the asymmetry that causes silent misses.

Transforming the query before retrieval addresses a class of failure that no amount of retriever tuning reaches.

Conversational rewriting

The most important one, and the most commonly missing.

User: How do I configure the cache?
Bot:  [explains cache configuration]
User: What about the timeout?

Embedding “What about the timeout?” retrieves documents about timeouts in general. The cache context — the thing that makes the question answerable — is nowhere in the query.

The fix is to rewrite the query as a standalone question using conversation history: What is the timeout setting for the cache?

Any multi-turn RAG system needs this. Without it, every follow-up question retrieves badly, and the symptom looks like a retrieval quality problem rather than a query problem.

Multi-query expansion

Generate several phrasings of the same question, retrieve for each, and merge the results.

How do I speed up my queries? becomes:

• query performance optimization
• slow database queries troubleshooting
• indexing for faster reads

Different phrasings land in different regions of the embedding space, so together they cover more ground than any single one. Merge with reciprocal rank fusion and deduplicate.

Cost is one extra cheap model call plus several retrievals. Usually worth it.

Decomposition

Compound questions retrieve poorly because no single document answers them.

How does our pricing compare to competitors and what did we change last quarter? is two questions. Embedded together, the vector sits between two topics and matches neither well.

Split, retrieve separately, combine the context. This is also the entry point to agentic RAG, where the model decides on the decomposition itself.

Hypothetical document embeddings

A counterintuitive technique that works well.

Instead of embedding the question, have a model write a hypothetical answer and embed that. The generated answer may contain factual errors — it does not matter, because it is only being used as a search key.

The reason it helps is that the hypothetical answer is phrased like a document, so it lands nearer real documents in embedding space than a question does. It attacks the question-statement asymmetry head-on.

Costs one generation before retrieval, and the answer needs no accuracy at all.

Step-back questions

For narrow, specific questions, retrieve on a more general version as well.

Does the v2.3 API support batch uploads? also retrieves on API upload capabilities. The specific version may not be documented separately, while the general capability is.

Useful when documentation is organized by concept rather than by version or variant.

Extracting filters

Queries often contain metadata constraints in natural language.

What did the Q3 report say about margins? implies a date and document-type filter. Extracting those into structured metadata filters, and retrieving only within the filtered set, is far more reliable than hoping semantic similarity captures “Q3.”

This applies to anything numeric or temporal, which vector search handles badly.

The cost

Every transformation adds latency and tokens before retrieval even begins. Some add a full generation.

Priorities, in order of value per unit of cost:

1. Conversational rewriting — mandatory for multi-turn, cheap, fixes a large failure class
2. Filter extraction — cheap, and vector search cannot do this at all
3. Multi-query expansion — good recall improvement, moderate cost
4. HyDE and decomposition — situational, one generation each

Use a small fast model for transformations. This is exactly the routing case: a cheap model rewrites, an expensive one answers.

Measuring

Transformations are easy to add and hard to evaluate by feel. The measurement that matters is unchanged: for real questions with known answers, was the correct chunk retrieved?

Run that with and without each transformation. Some will help on your corpus and some will not, and intuition is a poor guide to which.

What to remember

• Real queries are short, contextless, compound, and phrased unlike documents — transformation addresses failures no retriever tuning reaches.
• Conversational rewriting into standalone questions is mandatory for multi-turn systems.
• Multi-query expansion covers more embedding space; HyDE embeds a hypothetical answer to fix question-statement asymmetry.
• Extract metadata filters from natural language, since vector search cannot handle dates and numbers.
• Use a cheap model for transformation, and measure retrieval hit rate with and without.

Next: Contextual Retrieval