Agentic RAG

Let the model decide what to search for, read results, and search again. Multi-hop retrieval without building a graph.

Standard RAG retrieves once, then answers. One search, one shot.

Agentic RAG makes retrieval a tool the model can call repeatedly. It searches, reads what came back, decides whether that was enough, and searches again with a better query if not.

The difference matters for questions where the right search cannot be known in advance.

What the loop enables

Multi-hop. Which team owns the service that handles payments? Search one: find the payments service, learn its name. Search two: find who owns that service. The second query depends on the first result, so no single retrieval could have found the answer.

Self-correction. Retrieved chunks are irrelevant. A single-shot system answers from them anyway. An agent notices and reformulates.

Decomposition. A compound question gets split into separate searches, each retrieving well, rather than one query that matches nothing precisely.

Source selection. With several tools — documentation search, code search, ticket search, SQL — the model picks by question type instead of searching everything.

Sufficiency judgment. The model can conclude the corpus does not contain the answer and say so, rather than answering from the closest available passage. This is a real reduction in hallucination, and it is why the pattern is worth the cost.

The shape

An agent loop with retrieval tools:

loop:
  model decides: search, or answer
  if search: run it, append results
  if answer: return

Nothing structurally new. The design work is in the tool definitions and the stopping conditions.

What to get right

Separate tools, described contrastively. search_docs, search_code, search_tickets — each stating what it covers and when to prefer a sibling. Tool description quality determines whether the model searches the right place, and vague descriptions are the top failure here.

Compact results. Retrieved chunks accumulate in context and are resent every iteration. Return the minimum useful text and a source identifier; offer a separate tool to fetch full content when needed.

A hard iteration cap. Three to five searches. Without it, an agent that cannot find something searches indefinitely.

Repetition detection. Reissuing near-identical queries is the characteristic failure. Track what has been searched and surface it to the model.

Explicit permission to give up. Provide a way to conclude the answer is not in the corpus, and reward using it. Otherwise instruction-following pressure produces an answer from whatever was retrieved.

The cost

Every iteration is a full model call with the accumulated transcript, so cost grows quadratically with the number of searches. A five-search answer can cost far more than five times a single-shot one.

Latency compounds too: five sequential round trips, each including retrieval.

This makes routing important. Most questions are single-hop and should take the cheap path. Reserve the loop for questions that need it — either by classifying first, or by starting single-shot and escalating when the first retrieval scores poorly.

Compared with GraphRAG

Both address multi-hop questions, and they pay at opposite ends.

GraphRAG precomputes structure at index time. Expensive indexing, cheap queries, and it uniquely handles corpus-level questions like “what are the main themes.”

Agentic RAG computes the path at query time. No indexing cost, expensive queries, and it adapts to questions nobody anticipated.

For corpora that change often, or where query patterns are unpredictable, agentic retrieval is usually the better trade — it needs no rebuild. For repeated global questions over a stable corpus, precomputation wins.

Practical notes

Start single-shot. Add the loop when you can point to specific questions failing for lack of a second search. Many apparent multi-hop failures are actually chunking or hybrid search failures.

Log every search. The query sequence is your only way to diagnose why an answer was wrong — and it usually shows the problem immediately.

Combine with query rewriting. The model’s self-authored queries benefit from the same conversational rewriting a user query does.

Score the outcome, not the path. For evaluation, many search sequences reach the same correct answer; grading the path punishes valid variation.

What to remember

• Agentic RAG makes retrieval a tool in a loop — search, read, refine, search again.
• Enables multi-hop, self-correction, decomposition, source selection, and honest “not in the corpus” answers.
• Needs contrastive tool descriptions, compact results, an iteration cap, repetition detection, and permission to give up.
• Cost and latency grow with iterations — route so simple questions stay single-shot.
• Versus GraphRAG: pays at query time rather than index time; better for changing corpora and unanticipated questions.

Next: RAG over Images and Tables