Section 2.6: Transformation and Aggregation

A clean table is still not the table a model learns from. This section covers turning columns into features, element by element and across rows, summarizing groups and reshaping a table between long and wide, and combining tables by stacking them or joining a lookup onto them.

Section 2.5 fixed what was wrong inside each column. The features a model needs, here a model that predicts how long a listing stays on the market, are mostly built from several columns or several rows at once: a price against its neighborhood's average, a listing's distance to the center from a lookup table. Each of those steps can return a table that looks right and holds the wrong number of rows or the wrong values, so each lesson reconciles its counts before and after.

Lessons

1. Apply and Element-wise Transformation - vectorized features first, then map, apply and binning.
2. GroupBy, Aggregation, the MultiIndex and Reshaping - statistics per group, group-relative features, two-level indexes and grids.
3. Combining Datasets - Concat and Merge - stacking tables, and joining a lookup without multiplying rows.

By the end you will be able to:

• Build features with vectorized expressions, and reach for apply only where none exists.
• Summarize groups with named aggregation, and build group-relative features with transform.
• Reshape a table between long rows and a grid, and tell an empty cell from a zero.
• Merge a lookup table onto a dataset, and guard the merge so a duplicated key cannot multiply rows.