GQA: Training Generalized Multi-Query Transformer…

Grouped-query attention finds a practical middle ground between multi-head attention's quality and multi-query attention's decoding speed. Read it to see how sharing key-value heads within groups shrinks the KV cache, and how an existing multi-head checkpoint can be converted with only a small amount of additional pretraining.

Reading focus: How grouping query heads lets each group share one key head and one value head, reducing KV-cache memory and bandwidth without collapsing to a single shared pair. How mean-pooling the original key and value projection heads preserves more checkpoint information than selecting one head or initializing new heads randomly. Why five percent additional pretraining produced GQA models with quality close to multi-head attention and inference speed close to multi-query attention.

EMNLP 2023. Ainslie et al.. 35 min read, medium difficulty.