Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

ALiBi replaces position embeddings with fixed distance penalties in attention scores. Follow the arithmetic, compare equal-time training budgets, and learn why improved perplexity on longer blocks can come from fewer context resets rather than better use of distant facts.

Reading focus: How head-specific distance penalties change attention weights without imposing a hard context cutoff. Why shifted bias rows give the same softmax result and why the causal mask still matters. How the evaluation protocol separates stable extrapolation from evidence of long-range retrieval.

ICLR 2022. Ofir Press, Noah A. Smith, and Mike Lewis. 12 min read, medium difficulty.