How Models Are Aligned
Turning a text predictor into an assistant takes preference training. What that adds, what it costs, and why refusals feel inconsistent.
On this page
A pretrained model predicts text. It has no notion of being helpful, and asked a question it may produce more questions.
Alignment is the training that closes that gap. Two stages, and the second is the interesting one.
Stage 1: supervised fine-tuning
Train on examples of instructions paired with good responses. Ordinary fine-tuning on demonstration data.
This alone transforms behaviour — the model learns the request-response frame, that it occupies the assistant role, and roughly what a good answer looks like. Most of the “it acts like an assistant now” change happens here.
Its ceiling is that you can only demonstrate. Writing a perfect response to every kind of request is impossible, and for many prompts there is no single right answer — only better and worse ones.
Stage 2: learning from preferences
The insight that unlocked the rest: judging two responses is far easier than writing a perfect one.
So collect comparisons. Generate two responses to the same prompt, have a human indicate which is better, and repeat across many prompts. Now you have preference data rather than demonstration data.
RLHF uses this in three steps. Train a reward model — a separate network that takes a prompt and response and outputs a score, trained so preferred responses score higher. Then optimize the language model to maximize that score, using reinforcement learning (PPO is the classic choice). Constrain it with a penalty for drifting too far from the starting model, or it degenerates into whatever the reward model happens to overvalue.
That constraint is not optional. Without it, optimization finds degenerate outputs that score highly and read as nonsense.
DPO achieves a similar result without a separate reward model or RL machinery, deriving a loss that directly increases the relative likelihood of preferred responses. Simpler to implement and train, and now widely used. Several related variants exist.
RLAIF replaces human labels with model-generated ones, guided by a written set of principles. Scales far beyond human labelling capacity, and shifts the question to whether the judging model’s preferences are the ones you want.
What preference training actually shapes
Not knowledge — that came from pretraining. What it shapes:
Helpfulness. Answering the question asked rather than something adjacent.
Format habits. Structure, length, use of lists, whether responses open with a preamble.
Tone. Hedging, confidence, warmth.
Refusal behaviour. A base model has no concept of declining. Every refusal you have seen is a learned tendency from this stage.
Verbosity. Human labellers tend to prefer longer, more thorough-looking answers, so models trained on their preferences become more verbose. This is a well-documented bias, and it is why an aligned model often gives you four paragraphs where one would do.
The costs
Capability can be traded away. Aligning toward safe and predictable output narrows what a model will attempt. Sometimes visible on creative or unusual tasks, and it is why some practitioners still prefer base models for certain generation work.
Reward hacking. Optimization targets the reward model, not the underlying goal. Any gap between them gets exploited — outputs that score well while being worse. The reward model is a proxy, and proxies leak.
Refusals are tendencies, not rules. No rule engine is involved. Learned tendencies generalize imperfectly, which is why refusal behaviour can seem inconsistent across similar requests. Same reason a system prompt is a strong default rather than a boundary — both come from training patterns.
Preferences encode whose preferences. Labeller demographics, instructions given to them, and the principles written for AI feedback all shape the result. “Aligned” always means aligned to something specific.
Sycophancy. Models trained on human approval tend toward agreement, including agreeing with incorrect corrections. Directly traceable to the objective.
Why this matters practically
The assistant persona is a layer, not an essence. Underneath, it is still predicting tokens. Confident helpfulness is trained surface behaviour and carries no information about accuracy.
Verbosity is a training artifact, so asking for brevity explicitly is worth doing.
Instruction-following pressure contributes to hallucination. Answering was rewarded more than declining, which is why explicitly permitting “I don’t know” measurably changes behaviour — you are counteracting a trained bias.
Model updates change behaviour without changing version numbers. Alignment tuning shifts between releases, which is why evals need re-running.
What to remember
- Two stages: supervised fine-tuning on demonstrations, then preference training on comparisons.
- Judging two responses is easier than writing one perfectly — that asymmetry is the whole basis.
- RLHF uses a reward model plus RL with a drift penalty; DPO skips both; RLAIF replaces human labels with model ones.
- Shapes helpfulness, format, tone, refusals, and verbosity — not knowledge.
- Costs: capability tradeoffs, reward hacking, inconsistent refusals, encoded preferences, and sycophancy.