How RAG Works, Step by Step

Two phases: index your documents once, then retrieve and answer per question. Every stage has a failure mode worth knowing before you build.

RAG has two phases that run at completely different times. Most confusion about it comes from blurring them.

Indexing happens once, ahead of time, over your whole corpus. Retrieval and generation happen per question, in milliseconds.

Phase 1 · Indexing

Load and extract

Get text out of wherever it lives — PDFs, HTML, wikis, databases, code.

Underrated as a failure source. A PDF table extracted into scrambled text produces garbage that flows through every later stage. Retrieval quality frequently traces back to extraction quality, and this stage deserves more attention than it usually gets.

Chunk

Split documents into retrievable pieces. Whole documents are too large to send and too diffuse to match against a specific question.

Chunk boundaries directly determine what can be retrieved. Split a procedure across two chunks and neither contains the whole procedure. See How to Split Documents for RAG.

Embed

Convert each chunk into a vector with an embedding model. Chunks with similar meaning land near each other.

One rule with no exceptions: the same embedding model must be used for indexing and for queries. Different models produce incompatible spaces, and the failure is silent — you get results, they are just meaningless.

Store

Write vectors plus their text and metadata into a vector database, which is built to find nearest neighbors quickly among millions of vectors.

Store metadata generously: source document, section, date, permissions, URL. You need it for filtering and for citations, and adding it later means re-indexing.

Phase 2 · Retrieval and generation

Embed the query

Same model, same space. The question becomes a vector.

Note the mismatch this creates: questions and answers are phrased differently. “How do I reset my password?” and “Password reset procedure: navigate to Settings…” are semantically related but not identical in form. This asymmetry is a real source of retrieval misses, and it is what query rewriting techniques address.

Search

Find the chunks whose vectors are closest to the query vector, usually by cosine similarity. Return the top k — commonly 3 to 20.

Two refinements that matter in production:

Metadata filtering. Restrict by date, source, or permissions before or during search. This is also where access control gets enforced.

Hybrid search. Combine vector search with keyword search. Vectors handle meaning; keywords handle exact matches — error codes, product IDs, function names, proper nouns. Neither covers the other, and hybrid retrieval reliably outperforms either alone. This is one of the highest-value improvements available.

Rerank

Optionally, pass candidates through a more expensive model that scores each chunk against the query directly rather than comparing pre-computed vectors.

Retrieve 50 cheaply, rerank to the best 5. This is usually the single largest quality gain per unit of effort. See Why You Need a Reranker.

Assemble the prompt

Put retrieved chunks in context with clear instructions:

Answer the question using only the context below.
If the context does not contain the answer, say so.
Cite the source of each claim.

Context:
[1] {chunk_1}
[2] {chunk_2}

Question: {question}

Three details carry weight. Explicit permission to say “not found” substantially reduces invention. Numbered chunks make citation mechanically possible. Ordering matters, because middle-of-context material is used less reliably — put the strongest chunks at the beginning and end.

Generate

The model answers from provided text. This is reading comprehension rather than recall, which is why the whole architecture works.

Where it breaks

Failures are unevenly distributed, and knowing where to look saves time:

Diagnose retrieval before generation. Inspect what was actually retrieved for a failing question. If the right chunk was not in the retrieved set, no prompt change will help — and this is the majority of RAG failures.

What to remember

• Indexing (load → chunk → embed → store) runs once; retrieval → rerank → generate runs per query.
• The same embedding model must be used on both sides, or results are silently meaningless.
• Hybrid search and reranking are the two highest-value additions to a basic pipeline.
• Explicit permission to decline, numbered chunks, and edge placement all measurably improve output.
• Most failures are retrieval failures — always check what was retrieved first.

Next: How to Split Documents for RAG