Section 4.5: The Image Bridge, FashionMNIST

An image is a tensor of shape channels by height by width, one channel for a grayscale image (a picture with one brightness value per pixel), and a fully connected network can read it once it is flattened into a single row of 784 numbers. This section installs torchvision (PyTorch's library of image datasets and transforms), downloads FashionMNIST, 70,000 pictures of clothing in ten classes, looks at an image as a tensor and as a picture, trains a network of Section 4.3's shape on 50,000 of them with the loop of Section 4.3 and scores the test set once, at 86.8 percent. Then it rolls every test image two pixels to the right and watches the same network fall to 46 percent, and to 55 percent when the columns it moves into come in black, which is the failure convolution (sliding one small grid of weights over every position of an image) exists to answer, applied here to one image and trained in Module 5.

Section 4.4 completed the playbook on tabular data, where a row's columns have no neighbors and flattening costs nothing. An image breaks that, because a pixel means what it means beside its neighbors, and a network with one weight per pixel position has no way to know that a shirt moved. The failures are quiet: a test accuracy that reads as robustness and is not, a transform applied to the wrong object, statistics taken from one image, a validation score taken on the test set, and a shift that reads as a corrupted image when nothing is corrupted. None of them stops the program. This is the last section of Module 4, and it ends on an unsolved problem on purpose.

Lessons

  1. torchvision and Image Tensors - installing torchvision, FashionMNIST downloaded once, an image as a tensor of shape (1, 28, 28), looking at an image, normalizing pixels like any feature, and batches of images with a validation split.
  2. MLP on Images - Flatten and Train - 784 numbers per image, the network of 101,770 parameters, the loop on 50,000 images, a linear model on pixels for comparison, and the test set scored once with the classes it confuses.
  3. Where MLPs Fail - the Case for Convolution - a weight per pixel, shifting the image by a few pixels, why the network cannot know a shirt moved, what convolution does differently, and the cost in parameters.

By the end you will be able to:

  • Install torchvision, download an image dataset once, read an image's shape, dtype and channel, draw it, and normalize pixels with statistics from the training images alone.
  • Flatten a batch of images, train a fully connected network on them with a seeded loader and a carved validation set, and report one test score with the per-class view behind it.
  • Predict how that network behaves when an object shifts a few pixels, show it from a run, and state what structure flattening discarded.
  • Count what a fully connected layer costs an image and what a convolution costs, and say what a filter reads that a weight per pixel cannot.