RAG or Fine-Tuning?

Retrieval supplies knowledge; fine-tuning shapes behavior. Most people asking this question want retrieval, and a surprising number want neither.

The question is usually posed as a choice between two ways to make a model know your stuff. That framing is the problem, because they do different things.

RAG supplies knowledge at request time. Fine-tuning changes behavior permanently.

Once separated, most cases resolve immediately.

The dividing line

Ask what is actually missing.

Missing information → retrieval. The model does not know your refund policy, your codebase, or yesterday’s incident. Put it in the context. See Why RAG Exists.

Missing behavior → fine-tuning. The model knows plenty but will not reliably produce output in your required format, house style, or classification scheme. It needs its tendencies adjusted, and that lives in parameters.

The common error is reaching for fine-tuning to install facts. It works badly, for reasons worth being specific about.

Why fine-tuning is poor at facts

Facts land unreliably. Training nudges parameters toward patterns. Style is a pattern and transfers well. An individual fact seen a handful of times becomes a faint trace that reconstructs into something nearly right — the correct shape with a wrong detail. That is worse than not knowing, because it is indistinguishable from knowing.

Updates mean retraining. Documents change weekly. Retraining does not.

No provenance. A fine-tuned answer cannot cite a source, because the source dissolved into weights. For most business use, “where did this come from” is a hard requirement.

Access control vanishes. Once trained in, the model discusses content regardless of who is asking. With retrieval, permissions are enforced at query time.

What fine-tuning is genuinely good at

Format and structure reliability. When you need a specific output shape on every call and prompting gets you 95%, fine-tuning gets you higher — and cheaper, since the instructions no longer occupy tokens on every request.

Tone and style. Your organization’s voice is nearly impossible to specify in prose and straightforward to demonstrate with a few hundred examples.

Narrow classification at volume. A small fine-tuned model can match a much larger prompted one on a single well-defined task, at a fraction of the cost per call.

Cost reduction at scale. Moving a long prompt into the weights removes it from every request. At millions of calls, that is the whole argument.

Domain conventions. Specialized notation, medical shorthand, legal drafting patterns — form rather than fact.

The pattern: fine-tuning teaches how to respond, retrieval supplies what to respond about.

Try these first

Both options are frequently premature.

Better prompting. Most format complaints are underspecification. Free to test.

Few-shot examples. Three examples often match what people expect from fine-tuning, at a token cost. If examples work, you have your answer and no training run.

A stronger model. Sometimes the honest fix. Test with the identical prompt before building anything.

Structured output constraints. If the problem is JSON validity, schema-constrained generation guarantees it. No training required.

Reach for fine-tuning when prompting plus examples has plateaued and you have a measurable gap, not before.

Combining them

They compose cleanly, and the strongest production systems often use both: retrieval for current, permissioned, citable knowledge; light fine-tuning for the format and voice you need on every response.

Order matters. Get retrieval working first — it addresses the more common problem and is far cheaper to iterate on. Add fine-tuning only if a behavioral gap remains once knowledge is handled.

The practical cost picture

RAG costs engineering: extraction, chunking, embedding, a vector store, reranking. Ongoing cost is per-query tokens for retrieved context. Iteration is fast — change the index, see results immediately.

Fine-tuning costs a curated dataset, which is usually the real expense and always larger than anticipated. Then a training run, then evaluation, then a repeat every time the base model updates. Ongoing inference is cheaper, because the prompt shrinks.

Parameter-efficient methods like LoRA train a small number of additional parameters instead of the full model, cutting cost and making multiple task-specific variants practical. This is what most people mean by fine-tuning now.

What to remember

• RAG supplies knowledge; fine-tuning shapes behavior. Nearly every case resolves once you ask which is missing.
• Fine-tuning installs facts unreliably, needs retraining per update, and destroys provenance and access control.
• It excels at format reliability, tone, narrow classification, and cutting per-request cost at volume.
• Try prompting, few-shot, a stronger model, and output constraints first.
• They combine well — build retrieval first, add fine-tuning only for a remaining behavioral gap.

Next: What Is an AI Agent?