Section 2.5: Data Cleaning

Real data arrives with gaps, numbers stored as text, one label spelled three ways, dates as strings, repeated records and values no listing can have. This section covers finding and fixing each of them: missing values, data types and conversion, categorical, string and datetime columns, then duplicates, tidy structure, outliers and range validation, on housing.csv, whose gaps, repeats and extreme values were put there on purpose, and on small tables built in code for the problems the file does not have.

Section 2.4 loaded the table and selected from it. Selection takes the data as it is, and a model trained on the data as it is refuses the gaps or learns from them, and learns from the repeats and the impossible values along with everything else. Every fix in this section is also a decision that changes the numbers, so each lesson counts what a step removed or changed before it trusts the result.

Lessons

1. Missing Data - finding gaps, then dropping or filling them and counting what each choice costs.
2. Data Types and Conversion - what each dtype holds, casting, and parsing text into numbers.
3. Categorical, String and Datetime Data - cleaning text, ordered categories, and dates turned into features.
4. Duplicates, Tidy Structure, Outliers and Range Validation - removing repeats, checking values against rules, and flagging extremes.

By the end you will be able to:

• Count the missing values in every column and decide, column by column, whether to drop or fill them.
• Convert text columns into numbers and dates, and count what a conversion turned into missing values.
• Normalize text, build ordered categories, and turn dates into features a model can use.
• Remove duplicates on a chosen key, validate ranges, flag outliers, and reconcile every count against the source.