What Is a Neural Network?

A neural network is layers of simple units that each compute a weighted sum. Stack enough of them and they approximate almost any function. Here is how.

Once your data is a vector of numbers, you need something that turns that vector into a prediction. A neural network is one such machine — and it happens to be the one underneath every modern LLM. Despite the brain metaphor, it is far simpler than the name suggests: layers of tiny units, each doing grade-school arithmetic.

The neuron: a weighted sum

The basic unit, a neuron, takes several input numbers and produces one output number. It does this in two steps.

First, multiply each input by a weight and add them up, plus one extra number called a bias. If the inputs are x1, x2, x3:

sum = w1*x1 + w2*x2 + w3*x3 + b

That is a weighted sum. The weights say how much each input matters; the bias shifts the result up or down. These numbers — the weights and the bias — are what the network learns, and weights and biases covers them in depth.

Second, pass that sum through an activation function, a simple nonlinear squashing step. Hold that thought for a moment — the next section explains why it is not optional.

A single neuron computing a weighted sum is just a linear model. It can draw one straight boundary through the data. Useful, but limited.

Why one layer is not enough

A straight-line boundary cannot separate data that curls around itself. The classic example is XOR: label a point positive when exactly one of two inputs is on. No single straight line separates the positives from the negatives — they sit in opposite corners. One neuron fails at this, and so does any purely linear stack of neurons, because a chain of linear steps collapses back into a single linear step.

This is exactly where the activation function earns its place. Insert a nonlinear function between layers and the collapse stops: two layers can now bend the boundary, carving regions a straight line never could. Without nonlinearity, a hundred-layer network is no more powerful than one neuron. That is the entire reason activation functions exist.

Stacking into layers

A layer is a group of neurons that all read the same inputs and each produce one output. Put the input vector in, and the layer emits a new vector — one number per neuron.

Networks stack layers in sequence:

• The input layer is just your feature vector.
• Hidden layers sit in the middle. Each reads the previous layer’s output and produces its own. These are where the useful transformations happen.
• The output layer produces the final answer — one number for a regression, or one score per class for classification.

“Deep learning” is nothing more exotic than a neural network with several hidden layers. Depth lets early layers detect simple patterns and later layers combine them into complex ones — edges into shapes, shapes into objects; letters into words, words into meaning. Running data forward through these layers is the forward pass.

Why this works: universal approximation

There is a remarkable mathematical fact behind all this. A network with enough hidden units can approximate essentially any continuous function to any desired accuracy. Give it the right weights and it can mimic almost any mapping from input to output you care about.

Two cautions keep this from being magic. It says such weights exist, not that they are easy to find — finding them is the job of gradient descent and backpropagation, and it can fail. And “enough units” can mean an impractical number for a shallow network, which is part of why depth, not just width, matters in practice.

Still, this is the license for the whole enterprise: a simple, repeated unit, stacked and adjusted, can represent enormously complex relationships.

From this to a Transformer

Everything larger is a variation on this theme. The Transformer behind modern language models is, at heart, layers of these weighted-sum units with a clever mechanism — self-attention — for deciding which inputs each position should read. The neuron did not go away. There are just billions of them, arranged so that information flows between positions in a sequence. If you understand a layer of weighted sums with a nonlinearity, you understand the atom that everything else is built from.

What to remember

• A neuron computes a weighted sum of its inputs plus a bias, then applies a nonlinear activation.
• A single layer is linear and limited; nonlinear activations between layers are what let networks bend decision boundaries.
• Networks stack input, hidden, and output layers; “deep” just means several hidden layers, and depth builds simple patterns into complex ones.
• Universal approximation guarantees the right weights exist — but finding them is a separate, hard problem.
• The same weighted-sum unit scales up into Transformers and every modern LLM.

Next: Weights and Biases — the numbers a network actually learns.