---
title: "Qwen 3 From Scratch"
description: "Build Qwen3's attention: query heads that share key/value heads (GQA), RMSNorm on each query and key, and rotary positions, checked against the source."
canonical_url: "https://fanout.sh/ai/lessons/llm-from-scratch-qwen-3"
md_url: "https://fanout.sh/ai/lessons/llm-from-scratch-qwen-3.md"
access: "pro-preview"
---

# Qwen 3 From Scratch

Build Qwen3's attention: query heads that share key/value heads (GQA), RMSNorm on each query and key, and rotary positions, checked against the source.

- Access: Fanout Pro (public preview)
- Module: LLM From Scratch

## A glimpse of Qwen 3 From Scratch

Attention gives each token several ways to read the preceding tokens. Each attention head makes a query, compares it with saved keys, and uses the resulting weights to mix saved values.

Do all those heads need their own keys and values? The original dense Qwen3 models let several query heads share a key/value head. This is grouped-query attention, or GQA.

## Continue learning

Pro unlocks guided learning, complete notes, practical walkthroughs, exercises, and every AI Research lesson.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md