Activation Functions

Activation functions add the nonlinearity that lets deep networks learn curved patterns. Here is why they are non-negotiable and how ReLU and sigmoid differ.

Inside a neural network, each neuron computes a weighted sum and then runs it through an activation function — a small nonlinear step. That step looks minor and is the single reason deep networks work at all. Remove it and a hundred-layer network collapses into the power of one.

Why linear alone fails

A weighted sum is a linear operation. Stack two linear layers and the math simplifies: a linear function of a linear function is still just a linear function. You can prove it with algebra, but the consequence is what matters — without a nonlinear step between layers, adding layers buys you nothing. The whole tower reduces to a single straight-line model.

A straight-line model can only separate data a straight line can separate. It cannot learn “positive when exactly one input is on,” cannot trace the curved boundary between two interleaved spirals, cannot represent anything genuinely complex. Real data is full of curves and interactions. Linear models cannot bend.

The activation function inserts a bend after each layer. Now the layers stop collapsing, and stacking them compounds their power. Each layer can warp the space a little, and many small warps produce boundaries of arbitrary complexity. This is the mechanism behind the universal approximation property.

ReLU: the workhorse

The most common activation is almost embarrassingly simple. ReLU — rectified linear unit — outputs the input if it is positive and zero otherwise:

relu(x) = max(0, x)

Feed it 3.2, get 3.2. Feed it -1.5, get 0. That is the whole function. It is nonlinear (the kink at zero is the nonlinearity), and it is cheap — a single comparison, no expensive math. Cheapness matters when you evaluate it billions of times per forward pass.

ReLU also dodges a problem that plagued older activations: it does not squash large positive values, so the learning signal stays strong for active neurons even in deep networks. That property is a big part of why very deep networks became trainable, and it is why ReLU and its variants dominate the hidden layers of modern models.

Its one quirk: a neuron stuck outputting negatives always emits zero and can stop learning — the “dying ReLU” problem. Variants that let a small negative slope through exist to patch exactly this.

Sigmoid: squashing to a probability

Before ReLU, the sigmoid was standard. It squashes any input into the range 0 to 1 along a smooth S-curve: large negatives approach 0, large positives approach 1, and zero maps to 0.5.

That range makes sigmoid perfect for one specific job: turning a raw score into a probability. A binary classifier’s final neuron is almost always a sigmoid, so its output reads directly as “73% likely.” That connects straight to probability for AI and to how models express confidence.

But sigmoid fell out of favor for hidden layers. Its curve flattens hard at both ends, and where it is flat the learning signal nearly vanishes — the “vanishing gradient” problem. In a deep network, signals passing through many sigmoids fade to nothing, and early layers stop learning. This is one of the concrete reasons early deep networks were so hard to train, and why ReLU’s non-squashing behavior was such a practical breakthrough.

Choosing one

The rule of thumb is short. Use ReLU (or a variant) in hidden layers — cheap, keeps signal alive, trains well. Choose the output activation to match the task: sigmoid for a single probability, softmax for a probability distribution across many classes, and often no activation at all for regression, so the output can be any number.

Most other activations are refinements of these ideas — smoother kinks, learnable slopes — chosen when a specific problem calls for them. The concepts to hold onto are the two roles: nonlinearity in the hidden layers, and range-shaping at the output.

The connection to modern models

Every layer in a Transformer has an activation function in its feed-forward block, and the choice affects both speed and quality. When you read that a model uses a particular activation, it is a decision about exactly this: what nonlinear bend to apply after each weighted sum. The principle is unchanged from the two-neuron toy — nonlinearity is what lets depth mean something.

What to remember

• Without a nonlinear activation between layers, a deep network collapses into a single linear model that can only draw straight boundaries.
• ReLU (max(0, x)) is cheap, keeps the learning signal strong, and dominates hidden layers.
• Sigmoid squashes to 0-1, ideal for probabilities, but flattens at the ends and causes vanishing gradients in deep hidden layers.
• Match the output activation to the task: sigmoid for one probability, softmax for many classes, none for regression.

Next: Loss Functions — how a network measures exactly how wrong its output is.