Training vs Inference
One is an enormous one-time expense that changes the model. The other is what happens every time you send a prompt, and it changes nothing.
On this page
Two completely different activities get confused constantly, and the confusion produces bad expectations about cost, privacy, and what a model can learn.
Training builds the model. It happens once, costs enormously, and permanently changes the parameters.
Inference runs the model. It happens on every request, costs a little, and changes nothing at all.
Training
Training adjusts parameters so predictions improve. The loop:
- Feed in a batch of text.
- Have the model predict the next token at every position.
- Compare predictions to what actually came next.
- Compute how each parameter contributed to the error.
- Nudge every parameter slightly in the direction that reduces it.
- Repeat, billions of times.
Step 4 is backpropagation, and it is what makes training expensive: gradients must be computed and held for every parameter, requiring several times the memory of just running the model. Step 5 is gradient descent.
Frontier model training runs occupy thousands of specialized processors for weeks or months. Pretraining covers what actually happens during that period.
When it finishes, the parameters are frozen. That frozen set of numbers is the model.
Inference
Inference is a forward pass only. Text goes in, tokens come out, no parameter changes.
For each generated token: convert input to embeddings, push through every layer, produce a probability distribution, sample one token, append, repeat. See How Does an LLM Actually Write?.
No gradients, no memory of the computation, nothing retained. Send the same prompt twice and the second call has no idea the first happened.
The consequences
The model does not learn from your conversation. Corrections you offer work only because they now sit in the context window and get re-sent. Start a new conversation and every trace is gone. This is the single most common misunderstanding about how these systems work.
Knowledge is frozen at training. Whatever the training data contained is what the model retains. Nothing after the cutoff exists to it. RAG exists to work around this by supplying fresh information at inference time.
The cost structures are opposite. Training is a huge fixed cost paid once. Inference is a tiny marginal cost paid per request — but multiplied across millions of requests, total inference spending typically dwarfs training. This is why caching and efficient serving matter commercially.
Optimization targets differ. Training optimizes throughput: process as much data as possible. Inference optimizes latency: return this one answer fast. Different hardware profiles, different techniques. KV caching is purely an inference optimization and has no training equivalent.
Where fine-tuning sits
Fine-tuning is training — real gradient updates, real parameter changes — just far smaller in scale. Start from an existing trained model and continue training on a narrower dataset.
It permanently alters the model, produces a new set of weights, and requires training infrastructure. It is not a way to give a model new facts at request time; for that you want retrieval. RAG or Fine-Tuning? covers the choice.
What to remember
- Training changes parameters and happens once; inference reads parameters and happens per request.
- Inference changes nothing — no learning from your conversation, no memory between calls.
- Knowledge is frozen at the training cutoff; retrieval is how you supply anything newer.
- Training is a large fixed cost; inference is small per call but usually larger in total.