---
title: "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints"
description: "Grouped-query attention finds a practical middle ground between multi-head attention's quality and multi-query attention's decoding speed. Read it to see how sharing key-value heads within groups shrinks the KV cache, and how an existing multi-head checkpoint can be converted with only a small amount of additional pretraining."
canonical_url: "https://fanout.sh/daily/2026-08-03-grouped-query-attention"
md_url: "https://fanout.sh/daily/2026-08-03-grouped-query-attention.md"
last_updated: "2026-08-03"
access: "public"
---

# GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

Grouped-query attention finds a practical middle ground between multi-head attention's quality and multi-query attention's decoding speed. Read it to see how sharing key-value heads within groups shrinks the KV cache, and how an existing multi-head checkpoint can be converted with only a small amount of additional pretraining.

## Paper details

- Authors: Ainslie et al.

- Venue: EMNLP 2023

- Track: Inference

- Difficulty: Medium

- Reading time: 35 min

- Original paper: https://aclanthology.org/2023.emnlp-main.298/

## What you will learn

- How grouping query heads lets each group share one key head and one value head, reducing KV-cache memory and bandwidth without collapsing to a single shared pair.

- How mean-pooling the original key and value projection heads preserves more checkpoint information than selecting one head or initializing new heads randomly.

- Why five percent additional pretraining produced GQA models with quality close to multi-head attention and inference speed close to multi-query attention.

## Continue exploring

[Explore Grouped Query Attention](https://fanout.sh/knowledge-graph?node=ai-research%3Aconcept%3Agrouped-query-attention-gqa): Open Fanout's GQA node and connect the paper to multi-head attention, multi-query attention, KV-cache memory, checkpoint conversion, and faster decoding.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md