Why RAG Exists
A model cannot know your documents and will confidently invent answers about them. RAG is the fix, and the problem is worth understanding before the solution.
On this page
Ask a model about your company’s refund policy. It will answer. The answer will be fluent, structured, and entirely invented.
Three separate limitations converge on this failure, and RAG addresses all three. Understanding the problem first is worth the ten minutes, because otherwise the pipeline is just a diagram you will not remember.
Limitation 1 · It never saw your data
Training used public text. Your internal documents, your codebase, your customer records, your wiki were not in it. The model has no path to that information.
Limitation 2 · Knowledge is frozen
Training ended on a date. Nothing after it exists to the model — and it cannot tell you where that line falls, so you cannot rely on it to flag the gap.
Even for public information, anything recent is unavailable.
Limitation 3 · It answers anyway
This is what turns the first two from inconveniences into hazards.
A model optimized for plausible continuation does not decline when it lacks information. It produces something policy-shaped, in the register of a real policy, with the confidence of a real policy. There is no visible marker distinguishing a grounded answer from a fabricated one.
Missing knowledge plus fluent invention is the actual problem. Either alone would be manageable.
Why not just fine-tune?
The intuitive fix — train the model on your documents — works badly for knowledge.
Facts do not land reliably. Fine-tuning adjusts parameters toward patterns. It teaches form and style well. It teaches specific facts unreliably, and a fact that fails to land produces a confident near-miss rather than an absence.
Updates require retraining. Every document change means another training run. Documents change daily.
No provenance. A fine-tuned answer cannot cite its source, because the source dissolved into weights. For most business uses, “where did this come from” is a requirement.
Access control disappears. Once trained in, the model will happily discuss content the asking user is not permitted to see.
Why not just paste everything?
Context windows are large now. Why not send the whole corpus?
Corpora exceed windows. A large context still does not fit a document repository.
Cost scales with every token, on every request. Sending 100,000 tokens per question is expensive per call and absurd at volume.
Quality degrades. Attention spreads thin, and information in the middle of a long context is used less reliably. Relevant material buried in irrelevant material gets used worse than the same material alone.
Latency. Prefill on a huge prompt is slow, every time.
What RAG does
Retrieve the relevant part, then answer from it.
- Index your documents once, as embeddings.
- When a question arrives, find the passages closest to it in meaning.
- Put those passages in the prompt.
- Ask the model to answer from the provided text.
The reframing is the point. Instead of asking the model to recall, you ask it to read. Reading comprehension over supplied text is something these models are genuinely reliable at — far more reliable than recall.
Each limitation gets addressed: private data becomes available, updates are instant because you re-index rather than retrain, provenance exists because you know which passages you sent, and access control is enforceable at retrieval time.
What it does not fix
It cannot answer what is not in your corpus. Retrieval finds what exists. Missing information stays missing.
Retrieval quality caps everything. Retrieve the wrong passages and the model answers confidently from wrong passages. Most disappointing RAG systems have a retrieval problem, not a generation problem — see chunking and reranking.
Hallucination is reduced, not eliminated. Grounding helps substantially. It does not make invention impossible, particularly when retrieved passages are partially relevant.
Questions spanning many documents remain hard. “Summarize every complaint about shipping this quarter” is not a retrieval-a-few-passages problem.
Fine-tuning is still right for behavior. RAG supplies knowledge; fine-tuning shapes form, tone, and task-specific behavior. They solve different problems and combine well.
What to remember
- Three converging limitations: your data was never in training, knowledge is frozen, and the model answers regardless.
- Fine-tuning is poor for facts — unreliable landing, retraining per update, no provenance, no access control.
- Stuffing everything in context is expensive, exceeds capacity, and degrades quality.
- RAG converts recall into reading comprehension, which is what models are reliably good at.
- Retrieval quality is the ceiling on the whole system.