---
title: "Training Language Models to Follow Instructions with Human Feedback"
description: "InstructGPT shows why a larger language model is not automatically better at following a user's intent. Read it to see how demonstrations, preference rankings, a learned reward model, and reinforcement learning turn human judgments into a practical post-training pipeline for more helpful instruction following."
canonical_url: "https://fanout.sh/daily/2026-07-31-instructgpt"
md_url: "https://fanout.sh/daily/2026-07-31-instructgpt.md"
last_updated: "2026-07-31"
access: "public"
---

# Training Language Models to Follow Instructions with Human Feedback

InstructGPT shows why a larger language model is not automatically better at following a user's intent. Read it to see how demonstrations, preference rankings, a learned reward model, and reinforcement learning turn human judgments into a practical post-training pipeline for more helpful instruction following.

## Paper details

- Authors: Ouyang et al.

- Venue: NeurIPS 2022

- Track: AI Research

- Difficulty: Easy

- Reading time: 45 min

- Original paper: https://arxiv.org/abs/2203.02155

## What you will learn

- How supervised fine-tuning on human-written demonstrations gives a pretrained model an initial instruction-following policy.

- How rankings of model responses train a reward model that turns subjective human preferences into a reusable learning signal.

- Why optimizing that reward with PPO can make a 1.3B-parameter model more preferred than a 175B GPT-3 model, while still leaving important alignment limitations.

## Continue exploring

[Explore RLHF](https://fanout.sh/knowledge-graph?node=ai-research%3Aconcept%3Arlhf): Open Fanout's RLHF node and connect the paper to supervised fine-tuning, preference data, reward modeling, PPO, and instruction-following alignment.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md