Module 3
Module 3 - Classical ML: Why It Works, and Why the Score Lies
You will be able to fit a model to a table, score it on rows it never saw, and prove whether that score is real.
A model takes a row's features and returns an answer, a price, a probability, a group, and the numbers inside it, chosen from data by fitting, decide which answer. This module covers what a fit does and what it minimizes, the rows held back to measure a model, preprocessing that learns from the training rows alone, which features to build and which to drop, classification and what each kind of error costs, trees, forests and boosting, the checks that decide whether a score is real, and the structure a table holds when it has no target at all.
This is the part of the work whose output someone acts on: a price quoted to a seller, a listing flagged as likely to sell within 30 days. You hold listings back before anything learns from them, fit every scaling, encoding and filling step on the training rows alone, build features only from what a listing has when it is priced, choose the metric that answers the question actually asked, move a classifier's threshold when one kind of error costs more than the other, and test whether a better score is a better model or a luckier split.
Without it, a score is a number nobody has checked. A model scored on the rows it was fitted to is rewarded for remembering them. Leakage, information from the rows a model is scored on reaching the model before the score is taken, is quieter: a gap filled with a median the test rows helped compute, a feature computed from the very price it predicts, a setting picked by its test score. None of these raises an error, and each leaves a score that reads exactly like an honest one. Module 2 gave you a clean table. This module teaches you what a model fitted to it is worth, and how to prove it.
What you will build
One project, the capstone, at the end of the module.
The Leakage Hunt (PAID, $39, or $19 at the half price), where you follow the team's recipe to a model whose score is excellent and wrong, then find why, prove it and fix it. It starts from an empty directory and its bootstrap writes everything it starts from, so nothing you did or skipped earlier changes where you begin.
The sections
- What a Model Is, and What Fitting Means - 4 lessons. Installing scikit-learn and what a model is, the split and what a held-out set is for, the loss a fit minimizes, then regression metrics and reading a residual.
- Feeding a Model Honest Data - 3 lessons. Scaling and scikit-learn's fit-and-transform interface, encoding categorical features, then pipelines,
ColumnTransformer(a different preprocessing step for each kind of column) and filling gaps without leakage. - Feature Engineering - 2 lessons. What makes a feature good and building one from the columns you have, then selection, and why more features is not better.
- Classification, and the Cost of Being Wrong - 3 lessons. Logistic regression and the probability it returns, the confusion matrix and the metric that matches the question, then thresholds, ROC (receiver operating characteristic) curves, class weights and imbalanced classes.
- Why One Model Is Not Enough - 3 lessons. Decision trees and why they overfit (learn the noise in their training rows along with the pattern), with kNN (k-nearest neighbors) in the same lesson, random forests and why averaging works, then gradient boosting and why it is not averaging.
- Trusting Your Numbers - 3 lessons. Bias, variance, capacity and learning curves, cross-validation, then hyperparameter search (choosing the settings a model is built with).
- Is the Result Real? - 2 lessons. Sampling, confidence intervals and hypothesis testing, then offline gains against online reality, a score on stored data against what the model does once it is used.
- Structure Without a Target - 3 lessons. k-means clustering, eigenvectors and what PCA (principal component analysis) is doing, then interpreting the structure they find and reducing a table to fewer dimensions.
Prerequisites
Module 2. You are assumed to be able to load a CSV file into a pandas DataFrame and select from it by label and by position, find and fill missing values, bin, group and merge tables, compute over NumPy arrays, fit least squares with np.linalg.lstsq and read its residuals, and draw and save a chart with matplotlib. Module 1's Python is assumed as well, its classes above all, where Section 1.8 built the objects that carry their settings and learned state, the shape scikit-learn's fit and predict take, and so are Module 0's terminal, filesystem and git.
What the exam proves
PAID, $39, or $19 at the half price. That you can predict what a split, a fit, a pipeline or a metric returns, catch the step that runs clean and leaves a score that cannot be trusted, a score taken on the training rows, a statistic computed before the split, a setting chosen by the test score, and decide which metric answers a stated question and which of two scores to trust. The pass covers the capstone and the exam with every attempt, for the life of the product.
Set up your machine first
Lesson 3.1.1 installs scikit-learn into (bootcamp) and Lesson 3.5.3 installs XGBoost, so before Lesson 3.1.1 starts you need the machine Module 2 left: a Unix terminal (WSL on Windows), uv, Python 3.12.8 inside a virtual environment at ~/dev/bootcamp holding Module 2's five libraries, and VS Code opened on that folder, along with this module's own dataset, listings.csv. Module 3's setup page checks that machine, gives the one install line that brings (bootcamp) up to Module 2's end if you are starting here, says where the dataset goes and which lesson performs each install, and ends with a checklist that must pass before Lesson 3.1.1. Read it even if your prompt already shows (bootcamp).