Section 5.4: Text Transfer Learning and the Architecture Map

Section 5.2 adapted a vision network ImageNet had trained, and this section does the same for text. It installs transformers, Hugging Face's library of pretrained models, loads BERT-Mini, a four-layer encoder of 11,170,560 parameters pinned to one commit of its repository, and reads Banking77 through the tokenizer that came with it, WordPiece pieces in place of Section 5.3's whole words. A linear head on its frozen features reaches 0.7955 on the validation queries, under the bag of embeddings at 0.8760. Fine-tuning every weight with the loop you wrote, AdamW at a peak rate of 3e-4 with warmup and decay, reaches 0.8955, and the test queries score that choice once, at 0.8990. The last lesson lays every architecture of Modules 4 and 5 on one map and states where this bootcamp stops.

Section 5.3 trained every network from a seeded draw, and the bag won. Here the network arrives pretrained, and the failures are quiet once more. Hand BERT-Mini the word ids of Lesson 5.3.1's vocabulary and it raises nothing: it scores 0.0090 on the validation queries at a confidence near the one it shows on its own ids, and calls 1,289 of the 2,000 one intent, because the tokenizer belongs to the model. A rate of 1e-3 ends the fine-tune at 0.7065, under the frozen head's 0.7955, and a network read in training mode changes 100 answers between two passes over the same queries, its dropout still on.

Lessons

  1. Hugging Face - Loading a Pretrained Encoder - installing transformers, the Hub, a model id and a revision, WordPiece tokens, padding and the attention mask, the encoder and its load report, a BertLayer read as a post-norm block, mean pooling, frozen features with a linear head, and the wrong tokenizer.
  2. Fine-Tuning the Encoder With Your Own Loop - fine-tuning every weight, AdamW and decoupled weight decay, the learning rate on text, warmup and decay with LambdaLR, dropout at validation, testing once against the bag and the frozen head, and where the fine-tuned network wins.
  3. The Architecture Map and the Boundary - the assumption each architecture builds in, order and positions, what each shape costs, encoders and decoders, pretrained or from scratch, the map from a problem to the architecture to try first, and the boundary of this bootcamp.

By the end you will be able to:

  • Install transformers at a pinned version, load an encoder and its tokenizer at a pinned revision, and read what the tokenizer and the load report print.
  • Prove that a tokenizer and a model belong together, and catch the mismatch that raises no error.
  • Fine-tune a pretrained encoder with your own loop, a learning-rate schedule and evaluation mode, choose on validation and score the test queries once.
  • Choose an architecture for a stated problem, tabular rows, images, short text or long sequences, justify it by the assumption it builds in, and test the choice with a short run.