Section 3.6: Trusting Your Numbers

Every setting so far was chosen on one set of validation rows, and one set is one draw: the leaf size Lesson 3.5.1 chose on its validation rows never wins on ten other carves of the same training rows. This section separates the two ways a model is wrong, measures both by refitting it, and reads learning curves beside those refits, then replaces one validation set with five folds, and chooses settings on those folds while the test rows wait for one final score. Three lessons, no install.

Cross-validation and search fail as quietly as everything before them. An integer cv on rows sorted by price builds folds that each hold one price range, and the linear pipeline scores as low as -6.787 on them, a scorer for MAE comes back negative, a mean over five folds hides how far the folds spread, and a search's best score is the score that chose its winner: a search over seeds alone raised its reported score by 0.026 and its test score by less than 0.004. The section's rules are that every fold must be as new to the model as the rows it will meet in use, that cross-validating a pipeline refits every fitted step inside each fold, and that the test rows score the winner once.

Lessons

  1. Bias, Variance, Capacity and Learning Curves - bias and variance measured by refitting on bootstrap samples, capacity as the setting that trades one for the other, learning curves drawn from growing slices of the fit rows, and what their slopes say about more rows.
  2. Cross-Validation - k-fold cross-validation, folds as new as the rows a model will meet, every fitted step refitted inside each fold, stratified folds for a label, and two models compared on the same folds.
  3. Hyperparameter Search - settings chosen on folds, GridSearchCV and RandomizedSearchCV, a classifier's C searched by average precision and its threshold by the question's cost, and why the winner's score has been spent.

By the end you will be able to:

  • Measure a model's spread and the error of its averaged prediction by refitting it, and read a learning curve's gap and slope, beside those refits, as a case for more rows, more capacity or less.
  • Cross-validate a pipeline with an explicit splitter, and say why every fitted step is refitted inside each fold.
  • Stratify the folds for a label, and compare two models on the same folds.
  • Search settings with GridSearchCV and RandomizedSearchCV, tune a threshold for a cost, and report the test score rather than the search's best.