How AI Works
Build a real mental model of a transformer, from a single neuron up to attention.
For Students and the theoretically minded · 36 articles · ~46.3h · 4 stages
- 1
Machine learning from scratch
- What Is Machine Learning?Machine learning is programming by example instead of by rule. Here are the three ways a machine can learn from data, and when each one applies.ML Foundations
- How Data Becomes NumbersA model does arithmetic, so every input — text, images, categories — must first become a vector of numbers. Here is how that conversion works and why it matters.ML Foundations
- What Is a Neural Network?A neural network is layers of simple units that each compute a weighted sum. Stack enough of them and they approximate almost any function. Here is how.ML Foundations
- Weights and BiasesWeights and biases are the numbers a neural network learns. Weights set how much each input matters; biases shift the result. Everything a model knows lives here.ML Foundations
- The Forward PassThe forward pass is how a network turns an input into an output: layer by layer, each computing weighted sums and activations. This is inference in one word.ML Foundations
- Activation FunctionsActivation functions add the nonlinearity that lets deep networks learn curved patterns. Here is why they are non-negotiable and how ReLU and sigmoid differ.ML Foundations
- Loss FunctionsA loss function turns 'the model was wrong' into a single number to minimize. It defines what the model is trying to do — choose it carefully.ML Foundations
- Gradient DescentGradient descent is how models learn: measure the slope of the loss, step downhill, repeat. The single algorithm behind training almost every neural network.ML Foundations
- BackpropagationBackpropagation is how a network computes the gradient for every weight efficiently: apply the chain rule backward through the layers, reusing work as you go.ML Foundations
- Overfitting and GeneralizationA model that memorizes its training data fails on new data. Overfitting versus generalization is the central tension in all of machine learning.ML Foundations
- 2
The road to transformers
- Classic Text PreprocessingBefore models learned to read raw text, engineers cleaned it by hand: tokenizing, lowercasing, removing stop words, and stemming words to their roots.Classic NLP
- Bag of Words and N-gramsThe first way anyone turned text into numbers: count the words, ignore the order. Simple, surprisingly effective, and the baseline everything else is measured against.Classic NLP
- word2vec: Where Word Vectors BeganThe idea that changed NLP: represent each word as a dense vector learned from its neighbours, so that similar words land near each other and meaning becomes arithmetic.Classic NLP
- Recurrent Neural NetworksThe first neural network built to read sequences one step at a time, carrying a hidden state forward. How RNNs work, what they made possible, and the flaw that limited them.Classic NLP
- LSTM and GRU: Memory That LastsRecurrent networks that forgot too fast got a fix: gated memory cells. How LSTMs and GRUs decide what to keep, what to discard, and why they ruled NLP for a decade.Classic NLP
- Sequence to Sequence and Encoder-DecoderHow two recurrent networks were chained to turn one sequence into another — the architecture behind neural machine translation, and the bottleneck that led to attention.Classic NLP
- The Origin of AttentionBefore self-attention, there was Bahdanau attention: a 2014 fix for translation that let the decoder look back at the whole input. Where the idea came from and how it worked.Classic NLP
- Why Transformers WonRNNs read one step at a time and forgot too much. Transformers process everything at once and reach any distance in one hop. The three reasons the field switched, in full.Classic NLP
- 3
What an LLM is
- What Is a Token in AI?A token is the unit an AI model actually reads — not a word, not a letter. It explains your bill, your context limit, and why models miscount letters.Foundations
- What Is an LLM?Understand how language models generate text, how training differs from inference, and what an AI application adds.Foundations
- What Is an Embedding?Embeddings turn meaning into coordinates, which is what lets a machine compute with language. The foundation under search, RAG, and attention.Foundations
- Probability for AIModels do not output answers, they output probability distributions. Understanding distributions, softmax, and confidence explains how AI systems actually behave.ML Foundations
- How Does an LLM Actually Write?One token at a time, each conditioned on everything before it. The generation loop explains streaming, cost, and why models cannot revise.Foundations
- The Language Modeling ObjectivePredicting the next token is not a metaphor for what an LLM does — it is the exact training target. Here is how that objective is defined and why it needs no human labels.Foundations
- Logits and SoftmaxHow a model's raw output scores become a probability distribution over the next token. The softmax function, what temperature does to it, and why the numbers are relative, not absolute.Foundations
- 4
Inside the transformer
- How Does a Transformer Work?The architecture behind every current language model, traced from input tokens to output probabilities — one layer at a time.Internals
- Self-Attention, Explained VisuallyHow each word decides which other words matter to it. Queries, keys, and values — the mechanism at the center of every language model.Internals
- Why Multiple Attention Heads?One attention pass produces one blend. Running many in parallel lets a model track grammar, reference, and position at the same time.Internals
- How Does a Model Know Word Order?Attention is order-blind by construction. Position has to be injected separately, and how it is injected determines how far context can stretch.Internals
- The Feed-Forward Network in a TransformerHalf of every transformer layer is not attention — it is a small per-token network holding most of the model's parameters. What it computes and why factual knowledge appears to live there.Internals
- Normalization Layers: LayerNorm and RMSNormWhy every transformer block rescales its activations, how LayerNorm and RMSNorm do it, and why this unglamorous step is what makes deep models trainable at all.Internals
- Layers, Residuals, and DepthWhy models stack dozens of identical blocks, and the two small tricks that make deep stacks trainable at all.Internals
- The Output Projection: From Hidden State to Vocabulary LogitsThe final step of a transformer turns one hidden vector into a score for every token in the vocabulary. How that projection works, why it is the largest single matrix, and why models tie it to the embeddings.Internals
- What Is a KV Cache?Generating token 500 should not require recomputing tokens 1 through 499. The cache that prevents it, and the memory it costs instead.Internals
- Why Long Contexts Cost So MuchAttention compares every token with every token, so doubling the context quadruples the work. The constraint behind every context limit and price tier.Internals
- The Geometry of MeaningEmbedding spaces have structure: directions carry relationships, clusters carry categories, and the geometry behaves in ways that are useful and occasionally misleading.Internals