---
title: "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation"
description: "ALiBi replaces position embeddings with fixed distance penalties in attention scores. Follow the arithmetic, compare equal-time training budgets, and learn why improved perplexity on longer blocks can come from fewer context resets rather than better use of distant facts."
canonical_url: "https://fanout.sh/daily/2026-10-19-alibi-linear-attention-biases"
md_url: "https://fanout.sh/daily/2026-10-19-alibi-linear-attention-biases.md"
last_updated: "2026-10-19"
access: "public"
---

# Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

ALiBi replaces position embeddings with fixed distance penalties in attention scores. Follow the arithmetic, compare equal-time training budgets, and learn why improved perplexity on longer blocks can come from fewer context resets rather than better use of distant facts.

## Paper details

- Authors: Ofir Press, Noah A. Smith, and Mike Lewis

- Venue: ICLR 2022

- Track: AI Research

- Difficulty: Medium

- Reading time: 12 min

- Original paper: https://arxiv.org/abs/2108.12409v2

## What you will learn

- How head-specific distance penalties change attention weights without imposing a hard context cutoff.

- Why shifted bias rows give the same softmax result and why the causal mask still matters.

- How the evaluation protocol separates stable extrapolation from evidence of long-range retrieval.

## Continue exploring

[Review soft attention](https://fanout.sh/daily/2026-09-16-neural-translation-soft-attention): Start with normalized attention weights and weighted value mixtures before adding the distance penalty.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md