How AI Works
From machine learning basics to how a transformer works inside.
52 tutorials, spanning 6 of the 17 learning layers — ordered so each builds on the last.
1Entry
- The AI Jargon DecoderThirty AI terms, each in one sentence, each linked to a fuller explanation. Use it as an index when a word blocks you mid-article.
- In What Order Should You Learn AI?A dependency-ordered path through AI concepts, from tokens to agents — plus three routes depending on how deep you actually need to go.
- What AI Can and Cannot Do (Yet)A capability map: what current AI is genuinely good at, what it fails at, and the dangerous middle category where it looks competent but isn't.
- What Is AI, and How Do People Actually Use It?Forget the definitions. Here are the five things people actually do with AI every day, and what connects them.
2ML Foundations
- Probability for AIModels do not output answers, they output probability distributions. Understanding distributions, softmax, and confidence explains how AI systems actually behave.
- What Is Machine Learning?Machine learning is programming by example instead of by rule. Here are the three ways a machine can learn from data, and when each one applies.
- How Data Becomes NumbersA model does arithmetic, so every input — text, images, categories — must first become a vector of numbers. Here is how that conversion works and why it matters.
- What Is a Neural Network?A neural network is layers of simple units that each compute a weighted sum. Stack enough of them and they approximate almost any function. Here is how.
- Weights and BiasesWeights and biases are the numbers a neural network learns. Weights set how much each input matters; biases shift the result. Everything a model knows lives here.
- The Forward PassThe forward pass is how a network turns an input into an output: layer by layer, each computing weighted sums and activations. This is inference in one word.
- Activation FunctionsActivation functions add the nonlinearity that lets deep networks learn curved patterns. Here is why they are non-negotiable and how ReLU and sigmoid differ.
- Loss FunctionsA loss function turns 'the model was wrong' into a single number to minimize. It defines what the model is trying to do — choose it carefully.
- Gradient DescentGradient descent is how models learn: measure the slope of the loss, step downhill, repeat. The single algorithm behind training almost every neural network.
- BackpropagationBackpropagation is how a network computes the gradient for every weight efficiently: apply the chain rule backward through the layers, reusing work as you go.
- Overfitting and GeneralizationA model that memorizes its training data fails on new data. Overfitting versus generalization is the central tension in all of machine learning.
- Train, Validation, TestSplit your data into three parts with three different jobs. Confuse them and your model looks great in development and fails in the real world.
3Classic NLP
- Recurrent Neural NetworksThe first neural network built to read sequences one step at a time, carrying a hidden state forward. How RNNs work, what they made possible, and the flaw that limited them.
- LSTM and GRU: Memory That LastsRecurrent networks that forgot too fast got a fix: gated memory cells. How LSTMs and GRUs decide what to keep, what to discard, and why they ruled NLP for a decade.
- Sequence to Sequence and Encoder-DecoderHow two recurrent networks were chained to turn one sequence into another — the architecture behind neural machine translation, and the bottleneck that led to attention.
- The Origin of AttentionBefore self-attention, there was Bahdanau attention: a 2014 fix for translation that let the decoder look back at the whole input. Where the idea came from and how it worked.
- Classic Text PreprocessingBefore models learned to read raw text, engineers cleaned it by hand: tokenizing, lowercasing, removing stop words, and stemming words to their roots.
- Bag of Words and N-gramsThe first way anyone turned text into numbers: count the words, ignore the order. Simple, surprisingly effective, and the baseline everything else is measured against.
- Why Transformers WonRNNs read one step at a time and forgot too much. Transformers process everything at once and reach any distance in one hop. The three reasons the field switched, in full.
- word2vec: Where Word Vectors BeganThe idea that changed NLP: represent each word as a dense vector learned from its neighbours, so that similar words land near each other and meaning becomes arithmetic.
4Foundations
- Logits and SoftmaxHow a model's raw output scores become a probability distribution over the next token. The softmax function, what temperature does to it, and why the numbers are relative, not absolute.
- What Is a Token in AI?A token is the unit an AI model actually reads — not a word, not a letter. It explains your bill, your context limit, and why models miscount letters.
- What Is an Embedding?Embeddings turn meaning into coordinates, which is what lets a machine compute with language. The foundation under search, RAG, and attention.
- Where Does the Vocabulary Come From?Byte Pair Encoding in four steps, and why the resulting vocabulary is a permanent fossil of the text it was built from.
- What Is an LLM?Understand how language models generate text, how training differs from inference, and what an AI application adds.
- How Does an LLM Actually Write?One token at a time, each conditioned on everything before it. The generation loop explains streaming, cost, and why models cannot revise.
- Base Models vs Chat ModelsA pretrained model continues text; it does not answer questions. What instruction tuning adds, and why the assistant persona is a trained layer.
- The Language Modeling ObjectivePredicting the next token is not a metaphor for what an LLM does — it is the exact training target. Here is how that objective is defined and why it needs no human labels.
- Perplexity: A Language Model's Intrinsic MeasurePerplexity scores how surprised a model is by real text. What the number means, how it relates to loss, and why a lower score is not the same as a better assistant.
- Temperature and Top-p, ExplainedWhy the same prompt gives different answers, and how the two main sampling knobs actually reshape the model's choices.
- Training vs InferenceSeparate parameter updates, context, caches and application storage when thinking about training and inference.
- What Happens During Pretraining?Hide the next token, guess it, adjust on error, repeat a trillion times. How a simple objective at enormous scale produces broad capability.
- What Does "7B Parameters" Mean?Parameters are the adjustable numbers that hold everything a model learned. Why the count matters, and why it stopped being the whole story.
5Internals
- Decoding Strategies: Greedy Search and Beam SearchOnce a model produces a probability distribution, something has to turn it into a sequence. The deterministic strategies — greedy and beam search — and why the locally best token is not the globally best sentence.
- The Geometry of MeaningEmbedding spaces have structure: directions carry relationships, clusters carry categories, and the geometry behaves in ways that are useful and occasionally misleading.
- How Does a Transformer Work?The architecture behind every current language model, traced from input tokens to output probabilities — one layer at a time.
- Encoder, Decoder, and Encoder-Decoder ArchitecturesThree ways to arrange a transformer stack — encoder-only, decoder-only, and encoder-decoder — differ in one thing: which positions can see which. That choice determines what each is good for.
- The Feed-Forward Network in a TransformerHalf of every transformer layer is not attention — it is a small per-token network holding most of the model's parameters. What it computes and why factual knowledge appears to live there.
- Layers, Residuals, and DepthWhy models stack dozens of identical blocks, and the two small tricks that make deep stacks trainable at all.
- Normalization Layers: LayerNorm and RMSNormWhy every transformer block rescales its activations, how LayerNorm and RMSNorm do it, and why this unglamorous step is what makes deep models trainable at all.
- The Output Projection: From Hidden State to Vocabulary LogitsThe final step of a transformer turns one hidden vector into a score for every token in the vocabulary. How that projection works, why it is the largest single matrix, and why models tie it to the embeddings.
- Self-Attention, Explained VisuallyHow each word decides which other words matter to it. Queries, keys, and values — the mechanism at the center of every language model.
- What Is a KV Cache?Generating token 500 should not require recomputing tokens 1 through 499. The cache that prevents it, and the memory it costs instead.
- Why Multiple Attention Heads?One attention pass produces one blend. Running many in parallel lets a model track grammar, reference, and position at the same time.
- How Does a Model Know Word Order?Attention is order-blind by construction. Position has to be injected separately, and how it is injected determines how far context can stretch.
- Why Long Contexts Cost So MuchAttention compares every token with every token, so doubling the context quadruples the work. The constraint behind every context limit and price tier.
6Prompting
- What Is a Context Window?One token budget covering your prompt, the conversation history, retrieved documents, and the answer being generated. Everything competes for the same space.
- Why Do AI Models Hallucinate?Fluent, confident, wrong. Hallucination is not a bug in the system — it follows directly from what the system was built to do.