Language Models are Few-Shot Learners

GPT-3 asks whether one pretrained language model can take on a new task from text alone, without changing its weights. Read it to see how an instruction and zero, one, or several worked examples turn the prompt into a temporary task specification, why larger models used that context more effectively on many benchmarks, and where the result remained uneven or contaminated.

Reading focus: How zero-shot, one-shot, and few-shot prompting change the examples inside the context while the pretrained model weights remain fixed. Why next-token prediction can continue a demonstrated input-output pattern, how a finite context limits the teaching set, and why more examples are not always better. How the paper's CoQA and SuperGLUE results coexist with chance-level WiC, benchmark-overlap concerns, high compute cost, bias, and uncertainty about task learning versus task recognition.

NeurIPS 2020. Brown et al.. 35 min read, easy difficulty.