How AI Works
Build a real mental model of a transformer, from a single neuron up to attention.
- 1
Machine learning from scratch
- What Is Machine Learning? Machine learning is programming by example instead of by rule. Here are the three ways a machine can learn from data, and when each one applies. ML Foundations
- How Data Becomes Numbers A model does arithmetic, so every input — text, images, categories — must first become a vector of numbers. Here is how that conversion works and why it matters. ML Foundations
- What Is a Neural Network? A neural network is layers of simple units that each compute a weighted sum. Stack enough of them and they approximate almost any function. Here is how. ML Foundations
- Weights and Biases Weights and biases are the numbers a neural network learns. Weights set how much each input matters; biases shift the result. Everything a model knows lives here. ML Foundations
- The Forward Pass The forward pass is how a network turns an input into an output: layer by layer, each computing weighted sums and activations. This is inference in one word. ML Foundations
- Activation Functions Activation functions add the nonlinearity that lets deep networks learn curved patterns. Here is why they are non-negotiable and how ReLU and sigmoid differ. ML Foundations
- Loss Functions A loss function turns 'the model was wrong' into a single number to minimize. It defines what the model is trying to do — choose it carefully. ML Foundations
- Gradient Descent Gradient descent is how models learn: measure the slope of the loss, step downhill, repeat. The single algorithm behind training almost every neural network. ML Foundations
- Backpropagation Backpropagation is how a network computes the gradient for every weight efficiently: apply the chain rule backward through the layers, reusing work as you go. ML Foundations
- Overfitting and Generalization A model that memorizes its training data fails on new data. Overfitting versus generalization is the central tension in all of machine learning. ML Foundations
- 2
The road to transformers
- Classic Text Preprocessing Before models learned to read raw text, engineers cleaned it by hand: tokenizing, lowercasing, removing stop words, and stemming words to their roots. Classic NLP
- Bag of Words and N-grams The first way anyone turned text into numbers: count the words, ignore the order. Simple, surprisingly effective, and the baseline everything else is measured against. Classic NLP
- word2vec: Where Word Vectors Began The idea that changed NLP: represent each word as a dense vector learned from its neighbours, so that similar words land near each other and meaning becomes arithmetic. Classic NLP
- Recurrent Neural Networks The first neural network built to read sequences one step at a time, carrying a hidden state forward. How RNNs work, what they made possible, and the flaw that limited them. Classic NLP
- LSTM and GRU: Memory That Lasts Recurrent networks that forgot too fast got a fix: gated memory cells. How LSTMs and GRUs decide what to keep, what to discard, and why they ruled NLP for a decade. Classic NLP
- Sequence to Sequence and Encoder-Decoder How two recurrent networks were chained to turn one sequence into another — the architecture behind neural machine translation, and the bottleneck that led to attention. Classic NLP
- The Origin of Attention Before self-attention, there was Bahdanau attention: a 2014 fix for translation that let the decoder look back at the whole input. Where the idea came from and how it worked. Classic NLP
- Why Transformers Won RNNs read one step at a time and forgot too much. Transformers process everything at once and reach any distance in one hop. The three reasons the field switched, in full. Classic NLP
- 3
What an LLM is
- What Is a Token in AI? A token is the unit an AI model actually reads — not a word, not a letter. It explains your bill, your context limit, and why models miscount letters. Foundations
- What Is an LLM? A large language model is a machine trained to predict the next token. Here is why that simple objective produces something that looks like thinking. Foundations
- What Is an Embedding? Embeddings turn meaning into coordinates, which is what lets a machine compute with language. The foundation under search, RAG, and attention. Foundations
- Probability for AI Models do not output answers, they output probability distributions. Understanding distributions, softmax, and confidence explains how AI systems actually behave. ML Foundations
- How Does an LLM Actually Write? One token at a time, each conditioned on everything before it. The generation loop explains streaming, cost, and why models cannot revise. Foundations
- The Language Modeling Objective Predicting the next token is not a metaphor for what an LLM does — it is the exact training target. Here is how that objective is defined and why it needs no human labels. Foundations
- Logits and Softmax How a model's raw output scores become a probability distribution over the next token. The softmax function, what temperature does to it, and why the numbers are relative, not absolute. Foundations
- 4
Inside the transformer
- How Does a Transformer Work? The architecture behind every current language model, traced from input tokens to output probabilities — one layer at a time. Internals
- Self-Attention, Explained Visually How each word decides which other words matter to it. Queries, keys, and values — the mechanism at the center of every language model. Internals
- Why Multiple Attention Heads? One attention pass produces one blend. Running many in parallel lets a model track grammar, reference, and position at the same time. Internals
- How Does a Model Know Word Order? Attention is order-blind by construction. Position has to be injected separately, and how it is injected determines how far context can stretch. Internals
- The Feed-Forward Network in a Transformer Half of every transformer layer is not attention — it is a small per-token network holding most of the model's parameters. What it computes and why factual knowledge appears to live there. Internals
- Normalization Layers: LayerNorm and RMSNorm Why every transformer block rescales its activations, how LayerNorm and RMSNorm do it, and why this unglamorous step is what makes deep models trainable at all. Internals
- Layers, Residuals, and Depth Why models stack dozens of identical blocks, and the two small tricks that make deep stacks trainable at all. Internals
- The Output Projection: From Hidden State to Vocabulary Logits The final step of a transformer turns one hidden vector into a score for every token in the vocabulary. How that projection works, why it is the largest single matrix, and why models tie it to the embeddings. Internals
- What Is a KV Cache? Generating token 500 should not require recomputing tokens 1 through 499. The cache that prevents it, and the memory it costs instead. Internals
- Why Long Contexts Cost So Much Attention compares every token with every token, so doubling the context quadruples the work. The constraint behind every context limit and price tier. Internals
- The Geometry of Meaning Embedding spaces have structure: directions carry relationships, clusters carry categories, and the geometry behaves in ways that are useful and occasionally misleading. Internals