---
title: "Improving Language Understanding by Generative Pre-Training"
description: "GPT-1 made the two-stage transfer recipe concrete: learn a decoder Transformer from unlabeled books, serialize several supervised tasks into token sequences, then adapt the same network with a small output head. Follow the mechanism, the decisive no-pretraining ablation, and the limits of the 2018 evidence."
canonical_url: "https://fanout.sh/daily/2026-10-28-generative-pretraining-gpt"
md_url: "https://fanout.sh/daily/2026-10-28-generative-pretraining-gpt.md"
last_updated: "2026-10-28"
access: "public"
---

# Improving Language Understanding by Generative Pre-Training

GPT-1 made the two-stage transfer recipe concrete: learn a decoder Transformer from unlabeled books, serialize several supervised tasks into token sequences, then adapt the same network with a small output head. Follow the mechanism, the decisive no-pretraining ablation, and the limits of the 2018 evidence.

## Paper details

- Authors: Alec Radford et al.

- Venue: OpenAI 2018

- Track: AI Research

- Difficulty: Medium

- Reading time: 12 min

- Original paper: https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf

## What you will learn

- How next-token prediction creates a reusable training signal before the model sees downstream labels.

- Why task-specific input transformations let entailment, similarity, question answering, and classification share one decoder architecture.

- What the layer-transfer and no-pretraining ablations establish, and why the auxiliary language-model objective has a mixed result.

## Continue exploring

[Compare in-context adaptation](https://fanout.sh/daily/2026-09-03-gpt3-few-shot-learners): Contrast GPT-1's supervised gradient-based fine-tuning with GPT-3's later use of demonstrations inside the context window.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md