---
title: "Ring Attention with Blockwise Transformers for Near-Infinite Context"
description: "Ring Attention asks how exact Transformer context can grow across devices without gathering the whole sequence on each one. Read it to see how stationary query blocks, circulating key-value blocks, and compute-transfer overlap turn a device ring into aggregate context capacity while keeping per-host activation memory independent of total sequence length."
canonical_url: "https://fanout.sh/daily/2026-08-09-ring-attention"
md_url: "https://fanout.sh/daily/2026-08-09-ring-attention.md"
last_updated: "2026-08-09"
access: "public"
---

# Ring Attention with Blockwise Transformers for Near-Infinite Context

Ring Attention asks how exact Transformer context can grow across devices without gathering the whole sequence on each one. Read it to see how stationary query blocks, circulating key-value blocks, and compute-transfer overlap turn a device ring into aggregate context capacity while keeping per-host activation memory independent of total sequence length.

## Paper details

- Authors: Liu, Zaharia, and Abbeel

- Venue: ICLR 2024

- Track: Systems

- Difficulty: Hard

- Reading time: 50 min

- Original paper: https://arxiv.org/abs/2310.01889v4

## What you will learn

- Why blockwise attention removes the full score matrix but leaves a sequence-length cost in each layer's output activations.

- How each host keeps one query block while key-value blocks rotate through neighboring hosts and exact softmax statistics accumulate in any block order.

- Why the no-added-overhead claim requires c ≥ F/B, and why the paper's 6bch local-memory bound does not contain total sequence length s.

## Continue exploring

[Explore distributed attention](https://fanout.sh/knowledge-graph?node=ai-research%3Aconcept%3Aflash-attention): Open Fanout's FlashAttention node and connect exact blockwise attention to sequence partitioning, interconnect bandwidth, memory hierarchy, and long-context Transformer systems.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md