Overfitting and Generalization

A model that memorizes its training data fails on new data. Overfitting versus generalization is the central tension in all of machine learning.

The point of machine learning is never to do well on the examples you trained on. You already know the answers to those. The point is to do well on data you have never seen. A model that nails its training set but flops on new data has overfit, and avoiding that failure is the central discipline of the whole field.

Memorizing versus learning

Two students prepare for an exam. One memorizes the answers to last year’s paper. The other learns the underlying material. On last year’s paper, both score perfectly. On this year’s different paper, the memorizer collapses and the learner does fine.

That is exactly overfitting versus generalization. A model that generalizes has captured the real pattern in the data. A model that overfits has memorized the training examples, including their noise and accidents, and has no pattern to apply to anything new.

The trap is that overfitting looks like success while you train. The training loss keeps dropping, right down toward zero. If that were the only number you watched, you would celebrate a model that is quietly becoming useless on real data. This is why “100% on the training set” is a warning sign, not a trophy.

Seeing it happen

The clean way to catch overfitting is to hold out data the model never trains on and watch its loss there too. Early in training, both the training loss and the held-out loss fall together — the model is learning real patterns that help everywhere.

Then they split. Training loss keeps falling, but held-out loss bottoms out and starts to rise. That divergence is the signature of overfitting: past that point the model is no longer learning generalizable structure, it is memorizing the training set’s quirks at the expense of new data. The gap between training performance and held-out performance is the generalization gap, and a widening gap is the thing to watch for. Setting up that held-out data properly is what train/validation/test splits are for.

Why models overfit

Two ingredients drive it.

Too much capacity for the data. A model with far more parameters than the data can constrain has enough freedom to fit every training point exactly, noise included. With ten data points and a flexible enough model, it can thread a curve through all ten perfectly and learn nothing general. More capacity is not automatically better; it must be matched to how much real signal the data contains.

Too little data. The same model that overfits on a thousand examples may generalize fine on a million. More data leaves less room to memorize, because there is simply too much to memorize and the only compact way to fit it all is to find the actual pattern. This is a core reason the scale of training data matters so much for LLMs.

The opposite failure exists too. A model with too little capacity underfits — too simple to capture even the real pattern, so it does poorly on training and new data alike. The goal sits between: complex enough to learn the signal, constrained enough to ignore the noise.

Fighting overfitting

Several tactics push a model toward generalization, covered in depth under regularization but worth naming here:

• More data, including augmenting what you have with realistic variations. The most reliable fix when it is available.
• Regularization penalizes complexity, discouraging the model from relying too heavily on any one weight and nudging it toward simpler patterns.
• Early stopping halts training at the moment held-out loss starts rising, before memorization sets in.
• Simplify the model when capacity clearly outstrips the data.

The theme underneath all of them: deliberately make it harder for the model to memorize, so its best available option is to learn the general pattern instead.

The tension never fully resolves

You cannot set complexity to “correct” once and forget it. Push too hard against overfitting and you underfit; relax and you overfit again. Every model lives somewhere on this spectrum, and finding the right spot is an empirical search, not a formula. The formal version of this trade — how error splits into a part from being too simple and a part from being too sensitive — is the bias-variance tradeoff.

This tension is also why evaluation is its own hard discipline. A number on the training set tells you nothing you want to know. Only performance on held-out data, measured honestly, reveals whether a model has genuinely learned — and that is where the train/val/test split comes in.

What to remember

• The goal is generalization — performing well on unseen data — not low training loss.
• Overfitting is memorizing the training set (including its noise); it shows up as held-out loss rising while training loss keeps falling.
• It is driven by too much model capacity relative to the data, or too little data; underfitting is the opposite failure.
• Fight it with more data, regularization, early stopping, and simpler models — all ways of making memorization harder.
• The balance is an empirical search that never fully resolves, formalized as the bias-variance tradeoff.

Next: Train, Validation, Test — how to split your data so you can measure generalization honestly.