Gated Attention for Large Language Models:…

Gated Attention asks what happens when an attention head can decline to write its result. Read it to see why softmax's sum-to-one rule encourages sink behavior, how a query-dependent sigmoid gate after SDPA separates reading from writing, which ablations make that explanation credible, and where missing error bars, full-attention cost, and an incomplete long-context theory bound the result.

Reading focus: Why softmax can choose where to read without representing that no visible token is useful enough to write forward. Why the strongest gate is post-SDPA, query-dependent, head-specific, multiplicative, and free to approach zero. How to read the perplexity, training-stability, sink, and 128k RULER results without turning them into a universal speedup or proven causal theory.

NeurIPS 2025 Best Paper. Zihan Qiu et al.. 27 min read, medium difficulty.