Training & Architecture
Fine-tuning, adaptation, and the architectures behind modern models.
20 tutorials, spanning 2 of the 17 learning layers — ordered so each builds on the last.
12Training and Adaptation
- How Fine-Tuning Actually WorksContinue training an existing model on your own data. What it changes, what it cannot change, and why the dataset is the whole job.
- Model DistillationTrain a small model to imitate a large one. Often better than training the small model from scratch, and the reason cheap models got good.
- Hyperparameter TuningSome settings are learned during training; others you must choose before it starts. How to search the ones you choose without fooling yourself or wasting compute.
- LoRA and Parameter-Efficient Fine-TuningTrain a small number of extra weights instead of all of them. Nearly all the benefit at a fraction of the cost, plus swappable adapters.
- Reading Model BenchmarksBenchmark tables are marketing surfaces with real numbers in them. How to extract signal, and why your own twenty examples beat any leaderboard.
- Optimizers and Learning RateGradient descent says which way is downhill. The optimizer and learning rate decide how far to step and how to remember past steps. Why SGD gave way to Adam.
- What Is Quantization?Store weights with fewer bits. Large memory and speed savings for small quality loss, and the main reason big models run on small hardware.
- Regularization: Dropout and Weight DecayTechniques that make a model deliberately worse at fitting training data so it does better on data it has never seen. Why handicapping a model helps.
- How Models Are AlignedTurning a text predictor into an assistant takes preference training. What that adds, what it costs, and why refusals feel inconsistent.
- How Reasoning Models WorkModels trained to think at length before answering. The same token-equals-computation principle, moved from your prompt into the training.
- Synthetic DataGenerating training data with models. It works when you can verify or filter the output, and fails quietly when you cannot.
- Transfer Learning: Why Pretraining ReusesA model trained on one huge task can be repurposed for a different, smaller one with a fraction of the data. Why representations transfer, and when they do not.
13Advanced Architecture
- Attention Variants: Sparse, Linear, and Sliding WindowFull attention compares every token to every other, and the cost grows with the square of length. The variants that break that quadratic, and what each gives up.
- How Embedding Models Are TrainedA general model's internal vectors are not the same as a good embedding. What the contrastive objective optimizes, and why hard negatives make or break it.
- How Inference Serving WorksBatching, scheduling, and memory management. Why throughput and latency pull against each other, and what a serving system is actually doing.
- How Long Context Became PossibleContext windows grew from thousands to millions of tokens. Four changes made that happen, and none of them repealed the quadratic.
- Mixture of ExpertsHold many parameters, use a few per token. How models grew far larger without getting proportionally slower.
- Merging and Combining ModelsAverage the weights of two fine-tuned models and you often get one that does both jobs. Surprisingly effective, and nobody fully knows why.
- Speculative DecodingGuess several tokens with a small model, verify them all at once with the large one. Free speed, identical output.
- State Space ModelsAn alternative to attention with linear cost and constant memory. Where it wins, where it loses, and why hybrids are winning.