Section 3.2: Feeding a Model Honest Data

Section 3.1's model was only ever fitted on numeric columns with no gaps. A real table holds columns on very different scales, a build year in the thousands beside a distance of a few kilometers, neighborhoods and conditions stored as text, and gaps, and each needs a step that learns something from data before the model sees it: a scaler learns what to subtract and what to divide by, an encoder learns the categories, an imputer learns a fill value. Every one of those steps is a transformer (an object whose fit learns from the training rows and whose transform applies what it learned to any rows). The section covers scaling, encoding categories, filling gaps, and assembling every step with the model into one pipeline.

A step that learns can learn from the wrong rows. Lesson 3.1.2's rule is that nothing about a held-out row may reach the fit, and a scaler or a fill value computed on the whole table breaks it without an error. The quiet failures here are a transformer fitted again on the test rows, ordinal codes that put excellent below fair, a category the training rows never had, priced anyway once its error is switched off, and a column a ColumnTransformer drops without a word, and the section ends on an object that fits every step on exactly the rows it is given.

Lessons

  1. Scaling and the Transformer API - fitting on the training rows and transforming the rest, StandardScaler as Lesson 2.2.5's standardization, MinMaxScaler and RobustScaler, which models a scale changes, and keeping column names through pandas output.
  2. Encoding Categorical Features - why a model refuses text, one-hot encoding and the rank it costs beside an intercept, a category the training rows never had, ordinal codes in their real order, and a missing category kept as one of its own.
  3. Pipelines, ColumnTransformer and Leakage-Safe Imputation - fills learned from the training rows, the was-missing indicator, a different step for each group of columns, a pipeline of named steps from the raw table to a price, and which rows each step inside it learns from.

By the end you will be able to:

  • Fit a scaler, an encoder or an imputer on the training rows and apply it to any other rows with transform, never fit.
  • Reproduce StandardScaler with NumPy, and say which models its scaling changes and which it leaves as they were.
  • One-hot and ordinal encode categories, and choose between refusing a category the training rows never had and ignoring it.
  • Build one Pipeline that takes the raw table, gaps and text included, to a price, and explain why fitting it fits every step on exactly the rows it receives.