Qwen 3 From Scratch

LLM From Scratch

A glimpse of Qwen 3 From Scratch

Attention gives each token several ways to read the preceding tokens. Each attention head makes a query, compares it with saved keys, and uses the resulting weights to mix saved values.

Do all those heads need their own keys and values? The original dense Qwen3 models let several query heads share a key/value head. This is grouped-query attention, or GQA.

Continue learning

Pro unlocks guided learning, complete notes, practical walkthroughs, exercises, and every AI Research lesson.