Evaluation
Measuring whether a model, a pipeline, or an agent actually works.
15 tutorials, spanning 5 of the 17 learning layers — ordered so each builds on the last.
10Evaluation
- The Bias-Variance TradeoffPrediction error splits into two opposing parts. Too simple a model misses the pattern; too complex a model memorizes the noise. The tension between them shapes every modeling choice.
- Why Evaluation Is Harder Than TrainingTraining has one number to minimize. Evaluation has to decide what 'good' even means, and most of the ways that decision goes wrong are invisible until production.
- Benchmark Contamination and Goodhart's LawWhen a test becomes a target, it stops measuring. Two forces — leaked test data and metric-gaming — make published scores drift away from real capability.
- Precision, Recall, F1, and AccuracyFour numbers that describe a classifier, and the specific question each one answers. Pick the wrong one and you optimize for the wrong outcome.
- The Confusion Matrix: Why Accuracy DeceivesA 2x2 table of what a classifier got right and wrong, broken out by type. Every classification metric is derived from its four cells, and it shows why one number is never enough.
- Human Evaluation and Annotation AgreementFor subjective quality, humans are the ground truth. But humans disagree, so you have to measure whether your labels are consistent enough to trust.
- Regression Metrics: MSE, MAE, and R²When a model predicts a number instead of a label, you measure the size of its errors. MSE, MAE, and R² each answer a different question about those errors.
- Retrieval and Ranking MetricsSearch and RAG return a ranked list, not a single answer. Recall@k, precision@k, MRR, and nDCG each score a different thing about that list.
- ROC and AUC: Evaluation Without a ThresholdA classifier outputs a score, not a decision. ROC and AUC judge the score itself, before you pick any cutoff, and summarize ranking quality in one number.
- Statistical Significance: Why One Comparison DeceivesModel A scored 84%, model B scored 86%. Is B better, or did it get lucky on your test set? Significance testing separates a real difference from noise.
- Text Generation Metrics: BLEU, ROUGE, METEORHow do you score generated text against a reference? These metrics count word overlap, which is fast and cheap and systematically disagrees with human judgment.