Section 4.1: Tensors and the Derivative

A tensor is PyTorch's array: a grid of numbers of one dtype that knows which device holds it, the processor or an accelerator, and can keep a record of the operations that made it. That record is what the rest of the module runs on. A derivative (the slope of the loss at one value of a parameter) says which way to move that parameter and how steeply the loss falls there, and autograd replays the record backwards to compute one for every parameter at once. This section installs PyTorch, builds tensors from the listings Module 3 taught on, computes with them on a device you name once, computes the derivative of Lesson 3.1.3's loss by hand with NumPy, and then has autograd compute it and checks the two against each other.

Lesson 3.1.3 found the best price per square meter by trying 6,001 candidates and keeping the one with the smallest loss, and said that a scan is hopeless for thousands of parameters, which is why Module 4 searches without a grid. The derivative is how. The quiet failures start here too: a tensor built from a DataFrame column that writes through to the frame under a warning, a float32 statistic read to digits it does not have, a (2875, 1) column subtracted from a (2875,) vector that broadcasts into a table of eight million differences, a gradient that adds to the last one instead of replacing it, and a parameter reassigned into a tensor that never receives a gradient again. None of them stops the program.

Lessons

  1. Installing PyTorch and the Tensor - the processor-only install, what a tensor is, tensors from lists, NumPy and the listings, dtype, shape and the scalar tensor, and seeded random tensors.
  2. Tensor Operations and the Device Pattern - elementwise operations and type promotion, broadcasting, reductions with dim, X @ w + b, views, reshapes and in-place operations, and the one line that names the device.
  3. Derivatives and the Chain Rule - Computed by Hand - the derivative as the slope of the loss, measured by a centered difference, the rules for simple functions, the chain rule, the gradient of the squared-error loss, and one step against it.
  4. Autograd - Checked Against Your Own Derivative - requires_grad and the recorded graph, backward and .grad checked against Lesson 4.1.3, gradients that accumulate, the step under no_grad, and gradcheck.

By the end you will be able to:

  • Install PyTorch's processor-only build into (bootcamp), build float32 tensors from a DataFrame without sharing its memory, and read a tensor's dtype, shape and device.
  • Compute with tensors as with NumPy arrays, predict the shape of a broadcast or a matrix product, and write the device pattern so one line changes on a machine with an accelerator.
  • Compute the derivative of the squared-error loss by hand, check it against a centered difference, and take one step against the gradient with a learning rate you can defend.
  • Have autograd compute the same gradient, show it equals yours, zero it before the next pass, and update a parameter inside torch.no_grad().