The Language Modeling Objective

Predicting the next token is not a metaphor for what an LLM does — it is the exact training target. Here is how that objective is defined and why it needs no human labels.

Every capability a language model has traces back to a single training target: assign high probability to the token that actually came next. Not “understand the text,” not “be helpful” — just predict the next token. Understanding what that objective is, precisely, explains why these models can be trained on almost unlimited data and why their skills are a side effect rather than a goal.

The objective, stated exactly

Take any text. Cut it into tokens. Now walk through it left to right. At each position, the model sees everything before and must output a probability distribution over the whole vocabulary for what comes next.

For the sequence the cat sat, the model faces three prediction problems:

• Given the, predict cat
• Given the cat, predict sat
• Given the cat sat, predict whatever follows

The training signal is simple: the model gets credit proportional to how much probability it placed on the correct next token. Place 0.9 on cat and you are nearly right. Place 0.001 and you are heavily penalized. Formally this is maximum likelihood — tune the parameters so the actual training text is as probable as possible under the model.

Why it needs no labels

This is the property that made scale possible. Most machine learning needs labeled examples: a photo tagged “cat,” a review tagged “positive.” Someone has to produce those labels, which caps how much data you can use.

Language modeling has no such cap. The label for each position is just the next token, which is already sitting there in the text. The data labels itself. This is called self-supervised learning, and it means any text ever written is training data as-is — no annotation step, no human in the loop.

That is why the objective could be pointed at a substantial fraction of the written internet. The bottleneck moved from label availability to compute, which is a very different and much looser constraint. See How Models Are Pretrained for what running this objective at scale actually looks like.

Measuring the error

The model outputs a probability for the correct token; training needs to convert that into a number to minimize. The standard choice is cross-entropy loss: the negative logarithm of the probability assigned to the true next token.

The logarithm matters. If the model assigns probability 1.0 to the correct token, the loss is 0 — perfect. As the assigned probability drops toward 0, the loss climbs toward infinity. Being confidently wrong is punished far more than being uncertain. A model that hedges — spreading probability around — takes a moderate penalty; a model that bets everything on the wrong token takes a huge one.

Averaged over billions of positions, this loss is the single quantity training pushes down. Everything the model becomes is in service of that one descent.

The objective is trivial; being good at it is not

Here is the part worth sitting with. The objective is mechanical, but satisfying it well forces the model to build real machinery.

To predict the last word of The trophy would not fit in the suitcase because it was too — big or small? — the model must track what it refers to and reason about physical fit. To predict the closing token of a code block, it must track syntax opened many lines earlier. There is no shortcut. Low loss across varied enough text is only reachable by internalizing grammar, facts, and logic.

This is why capability is described as emergent. Nobody trained the model to do arithmetic. Arithmetic is simply one of the things you need in order to predict the next token in text that contains arithmetic. The full argument is worth reading, but the mechanism is this objective doing the forcing.

Teacher forcing

One subtlety about training that surprises people. During training the model does not generate freely and get graded on its own output. At every position it is shown the real previous tokens from the training text, then asked for the next one.

This is teacher forcing. It means every position can be trained in parallel — the model never has to wait for its own earlier guesses — which is a large part of why training is efficient. The catch is that training and generation differ: at generation time the model must consume its own previous outputs, errors and all. That gap is one reason generation can drift in ways training never showed.

What the objective does not specify

The objective says “predict the next token in the training data.” It says nothing about being truthful, helpful, or safe. A model trained purely this way imitates its data, including its data’s contradictions and mistakes. It will happily continue a question with more questions if that is what its text tended to do.

Making a model behave like an assistant is a separate stage layered on top, covered in Base Models vs Chat Models. The language modeling objective builds the raw competence; it does not aim it.

What to remember

• The objective is to maximize the probability of the actual next token — maximum likelihood, nothing more.
• It is self-supervised: the label is the next token, already in the text, so any writing is training data without annotation.
• Cross-entropy loss penalizes confident wrong predictions sharply, and its average is what training minimizes.
• Broad capability is emergent — the forced consequence of predicting well across varied text, not a separate goal.
• The objective builds competence but does not aim it; assistant behavior is a later training stage.

Next: Logits and Softmax — how the model’s raw scores become the probability distribution this objective is scored against.