Ring Attention with Blockwise Transformers for…

Ring Attention asks how exact Transformer context can grow across devices without gathering the whole sequence on each one. Read it to see how stationary query blocks, circulating key-value blocks, and compute-transfer overlap turn a device ring into aggregate context capacity while keeping per-host activation memory independent of total sequence length.

Reading focus: Why blockwise attention removes the full score matrix but leaves a sequence-length cost in each layer's output activations. How each host keeps one query block while key-value blocks rotate through neighboring hosts and exact softmax statistics accumulate in any block order. Why the no-added-overhead claim requires c ≥ F/B, and why the paper's 6bch local-memory bound does not contain total sequence length s.

ICLR 2024. Liu, Zaharia, and Abbeel. 50 min read, hard difficulty.