---
title: "Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free"
description: "Gated Attention asks what happens when an attention head can decline to write its result. Read it to see why softmax's sum-to-one rule encourages sink behavior, how a query-dependent sigmoid gate after SDPA separates reading from writing, which ablations make that explanation credible, and where missing error bars, full-attention cost, and an incomplete long-context theory bound the result."
canonical_url: "https://fanout.sh/daily/2026-09-30-gated-attention-language-models"
md_url: "https://fanout.sh/daily/2026-09-30-gated-attention-language-models.md"
last_updated: "2026-09-30"
access: "public"
---

# Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free

Gated Attention asks what happens when an attention head can decline to write its result. Read it to see why softmax's sum-to-one rule encourages sink behavior, how a query-dependent sigmoid gate after SDPA separates reading from writing, which ablations make that explanation credible, and where missing error bars, full-attention cost, and an incomplete long-context theory bound the result.

## Paper details

- Authors: Zihan Qiu et al.

- Venue: NeurIPS 2025 Best Paper

- Track: AI Research

- Difficulty: Medium

- Reading time: 27 min

- Original paper: https://arxiv.org/abs/2505.06708

## What you will learn

- Why softmax can choose where to read without representing that no visible token is useful enough to write forward.

- Why the strongest gate is post-SDPA, query-dependent, head-specific, multiplicative, and free to approach zero.

- How to read the perplexity, training-stability, sink, and 128k RULER results without turning them into a universal speedup or proven causal theory.

## Continue exploring

[Review QKV and attention sinks in Fanout](https://fanout.sh/inference-eng/curriculum/02-02): Connect the paper's write gate to the attention-head interface, causal masking, first-token sinks, and the residual stream it modifies.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md