Improving Language Understanding by Generative Pre-Training

GPT-1 made the two-stage transfer recipe concrete: learn a decoder Transformer from unlabeled books, serialize several supervised tasks into token sequences, then adapt the same network with a small output head. Follow the mechanism, the decisive no-pretraining ablation, and the limits of the 2018 evidence.

Reading focus: How next-token prediction creates a reusable training signal before the model sees downstream labels. Why task-specific input transformations let entailment, similarity, question answering, and classification share one decoder architecture. What the layer-transfer and no-pretraining ablations establish, and why the auxiliary language-model objective has a mixed result.

OpenAI 2018. Alec Radford et al.. 12 min read, medium difficulty.