Section 3.4: Classification, and the Cost of Being Wrong

Sections 3.1 to 3.3 predicted a price. This section predicts a yes or a no: whether a listing sells within 30 days of going up, sold_in_30_days, true for about one listing in five. A classifier (a model whose answer is one of a few classes, here sold or not) returns a probability for each listing, and a threshold (the cut that turns a probability into a yes or a no) makes the answer. The section covers logistic regression and the probability it returns, the confusion matrix and the metrics built from its cells, and the threshold, the curves and the class weights a label this unbalanced needs.

A classifier fails quietly in its own ways. A rule that calls every listing unsold scores about 0.79 accuracy and finds no fast sale at all, and the logistic model at its default threshold scores about the same while calling three listings of 600 fast sales. A confusion matrix read with its rows and columns swapped, precision asked of a model that predicted no yes, a metric picked after seeing which one looks best, a threshold chosen on the test rows and a weighted model's probabilities read as chances all print a number without an error. The section's rule is that the question names its metric before any model is scored, and the threshold is a setting chosen on validation rows.

Lessons

  1. Logistic Regression and the Probability It Returns - why a straight line fitted to a yes or a no fails, the logistic function from a score to a probability, LogisticRegression in the section's pipeline, its weights read as odds, and C, which divides the cost the fit pays on large weights.
  2. The Confusion Matrix, and Choosing a Metric That Matches the Question - accuracy against the majority baseline, the four cells of the confusion matrix, precision, recall and F1, and the metric each of two questions names before any model is scored.
  3. Thresholds, ROC, Class Weights and Imbalanced Classes - the threshold as a choice, precision-recall and ROC curves, what AUC measures and the question it cannot answer, class weights and the probabilities they move, and a threshold chosen on validation rows.

By the end you will be able to:

  • Fit LogisticRegression in a pipeline, read predict_proba and predict, and read a weight as a change in log-odds and a factor on the odds.
  • Build and read a confusion matrix, and compute accuracy, precision, recall and F1 from its cells by hand.
  • Name the metric a question needs before scoring a model, and say why accuracy says little on a label this unbalanced.
  • Read precision-recall and ROC curves, choose a threshold on validation rows, and say what class weights do to a probability.