Gradient Descent

Gradient descent is how models learn: measure the slope of the loss, step downhill, repeat. The single algorithm behind training almost every neural network.

A loss function tells you how wrong the model is. Gradient descent is how the model gets less wrong: it figures out which way to nudge each weight to lower the loss, and takes a small step that way. Repeat millions of times and a network of random numbers becomes a working model. This one algorithm trains nearly every neural network, including every LLM.

The valley metaphor

Imagine the loss as a landscape. The model’s current weights place it somewhere on this terrain, at some height — the current loss. High ground is bad (large loss), low ground is good. Training means walking downhill toward a valley.

The catch: you cannot see the whole landscape. It has billions of dimensions, one per parameter, and you only know your local surroundings. So how do you head down when you cannot see the valley? You feel the slope right where you stand and step in the steepest downhill direction. That is gradient descent in one sentence.

The gradient is the slope

The gradient is the mathematical slope of the loss with respect to the weights. For each weight, it answers a precise question: if I increase this weight a little, does the loss go up or down, and how sharply?

• A large positive gradient for a weight means “increasing this weight raises the loss a lot” — so decrease it.
• A large negative gradient means “increasing it lowers the loss a lot” — so increase it.
• A near-zero gradient means “this weight barely affects the loss right now” — leave it mostly alone.

The gradient bundles one such number for every weight in the model — billions of them — into a single direction pointing uphill. To go down, you step the opposite way. Where these numbers come from across a deep network is the job of backpropagation; gradient descent is what you do with them once you have them.

The update rule and the learning rate

Each step follows one rule: move every weight a little in the downhill direction.

new_weight = old_weight - learning_rate * gradient

The learning rate is the size of the step, and it is the most important knob in training. Too small and learning crawls, taking forever and getting stuck in shallow dips. Too large and you leap clean over the valley, bouncing around or flying off to infinity — the loss becomes not-a-number and training dies.

There is no universally correct value; it is tuned per problem, and picking it well is much of the art of training. Modern optimizers adjust it automatically as they go, a topic taken up in optimizers and learning rate.

One step at a time, in batches

You could compute the gradient over the entire training set before each step. For a large dataset that is impossibly slow — one step might require reading billions of examples.

Instead, training uses stochastic gradient descent: estimate the gradient from a small random batch of examples, take a step, grab the next batch, repeat. Each estimate is noisy, but the noise averages out over many steps, and you get thousands of cheap steps instead of a few expensive ones. The noise even helps — it can jostle the model out of shallow bad spots. Batch size becomes a real knob: bigger batches give steadier gradients but cost more memory and compute per step.

One full pass through the training data is an epoch. Training runs for many epochs, seeing every example repeatedly, each time nudging the weights a little lower.

Why it does not need to be perfect

The loss landscape of a big network is not a single clean valley. It is a rugged surface with countless dips, ridges, and flat plains. Gradient descent is not guaranteed to find the deepest valley — the global minimum — and in fact it rarely does.

This turns out to be fine. In high-dimensional networks, the many local valleys tend to be nearly as good as each other, so landing in a decent one produces a strong model. Chasing the theoretical best is unnecessary; “good enough, reliably” is the actual target. This gap between “provably optimal” and “works well in practice” is a recurring theme in the field, and gradient descent is its clearest example.

The complete training loop

Put the pieces together and you have how models learn:

1. Forward pass: run a batch through the network to get predictions.
2. Compute loss: measure how wrong those predictions are.
3. Backward pass: use backpropagation to compute the gradient of the loss for every weight.
4. Update: nudge each weight downhill by learning rate times its gradient.
5. Repeat for the next batch, for many epochs.

Every trained model you have used — every LLM, every image classifier — is the product of this loop run at enormous scale. The whole of pretraining is this loop applied to next-token prediction across trillions of tokens.

What to remember

• Gradient descent minimizes the loss by repeatedly stepping downhill in weight space.
• The gradient is the slope of the loss for every weight; stepping opposite to it lowers the loss.
• The learning rate sets step size — too small crawls, too large diverges — and is the key knob.
• Stochastic gradient descent uses small random batches for many cheap, noisy steps instead of few expensive ones.
• It finds a good-enough valley, not the perfect one, and for large networks that is enough.

Next: Backpropagation — how the gradient for every weight is actually computed.