Optimizers and Learning Rate

Gradient descent says which way is downhill. The optimizer and learning rate decide how far to step and how to remember past steps. Why SGD gave way to Adam.

Gradient descent tells you which direction reduces the loss. It does not tell you how big a step to take, or whether to trust this step given the last hundred. Those decisions belong to the optimizer and its most important knob, the learning rate — and getting them wrong wastes more training runs than any other single cause.

Learning rate: the one knob that matters most

The learning rate scales every update. Gradient says “go this way”; learning rate says “go this far”.

The failure modes sit on either side of it:

• Too high. Steps overshoot the minimum, bouncing across the valley or diverging entirely — the loss shoots up and turns to garbage. This is the single most common training failure.
• Too low. Training crawls. It will get there eventually, but you burn compute doing it, and it can stall in a shallow spot long before reaching a good solution.

There is no universal right value because it depends on the loss landscape, the model, and the data. This is why learning rate is the first hyperparameter anyone tunes, and why fine-tuning demands a far smaller one than pretraining — you are nudging an already-good model, not carving one from scratch, and a large step would destroy what it knows.

Schedules: change the step over time

A fixed learning rate is rarely ideal. Early on you want large steps to cover ground; later you want small ones to settle precisely. So the rate changes on a schedule.

Warmup. Start tiny and ramp up over the first steps. Early gradients are noisy and a big initial step can send an untrained model somewhere unrecoverable. Warmup avoids that.

Decay. After warmup, shrink the rate gradually — linearly, or on a cosine curve — so the model takes finer steps as it nears a good region.

The shape of this curve is itself something you tune. The instinct to remember: big steps to explore, small steps to converge.

Momentum: remember where you were going

Plain gradient descent is memoryless — each step uses only the current gradient. That makes it jittery. Noisy gradients from a small batch send it zig-zagging.

Momentum fixes this by accumulating a running average of past gradients, like a ball rolling downhill that builds speed in a consistent direction and coasts through small bumps. Directions that keep pointing the same way accelerate; directions that flip back and forth cancel out. The result is faster, smoother progress along the real slope.

From SGD to Adam

This is the arc worth understanding, because it explains why one optimizer became the default.

SGD (stochastic gradient descent). The baseline: step in the gradient direction, scaled by the learning rate, using a small random batch each time. Add momentum and it works well — but it uses a single learning rate for every parameter, and it is sensitive to getting that rate right.

The problem: different parameters need different step sizes. A parameter that rarely receives a strong gradient should take bigger steps when it finally does; one that is constantly pushed hard should take smaller ones. A single global rate cannot do both.

Adaptive methods solve this by giving each parameter its own effective step size, based on the history of gradients it has seen. Parameters with large, frequent gradients get damped; quiet parameters get amplified.

Adam combines the two good ideas: momentum (a running average of gradients) and per-parameter adaptive scaling (a running average of gradient magnitudes). The payoff is practical — it works reasonably well across many problems with far less learning-rate tuning than SGD needs. That robustness is why it became the default for training large models, where each failed run is enormously expensive.

Adam is not strictly better — well-tuned SGD with momentum can generalize as well or better on some tasks. Adam wins on not needing to be perfectly tuned, which matters most exactly when you cannot afford many attempts.

Reading a training curve

The optimizer and learning rate show their symptoms in the loss curve, and learning to read it saves runs:

• Loss explodes or turns to NaN → learning rate too high. Cut it.
• Loss barely moves → too low, or a bug upstream in the gradients.
• Loss spikes then recovers repeatedly → rate slightly too high, or batches too small and noisy.
• Loss plateaus early → may need a schedule change, or you have hit the limit of the model or data.

Most of tuning is diagnosis from this curve, not theory.

What to remember

• Learning rate scales every step — too high diverges, too low crawls, and it is the first thing to tune.
• Schedules (warmup then decay) give big exploratory steps early and fine steps near convergence.
• Momentum averages past gradients to smooth out noise and accelerate consistent directions.
• Adam combines momentum with per-parameter adaptive step sizes; it wins by needing far less tuning, which matters when runs are expensive.
• Diagnose problems from the loss curve — explosions mean too-high rate, flatness means too-low or a bug.

Next: Regularization