Training Language Models to Follow Instructions…

InstructGPT shows why a larger language model is not automatically better at following a user's intent. Read it to see how demonstrations, preference rankings, a learned reward model, and reinforcement learning turn human judgments into a practical post-training pipeline for more helpful instruction following.

Reading focus: How supervised fine-tuning on human-written demonstrations gives a pretrained model an initial instruction-following policy. How rankings of model responses train a reward model that turns subjective human preferences into a reusable learning signal. Why optimizing that reward with PPO can make a 1.3B-parameter model more preferred than a 175B GPT-3 model, while still leaving important alignment limitations.

NeurIPS 2022. Ouyang et al.. 45 min read, easy difficulty.