---
title: "Self Attention from Scratch"
description: "Give each token a query, a key and a value, turn the scores into weights with softmax, mix the values, then add a causal mask so no token reads ahead."
canonical_url: "https://fanout.sh/ai/lessons/transformers-self-attention"
md_url: "https://fanout.sh/ai/lessons/transformers-self-attention.md"
access: "pro-preview"
---

# Self Attention from Scratch

Give each token a query, a key and a value, turn the scores into weights with softmax, mix the values, then add a causal mask so no token reads ahead.

- Access: Fanout Pro (public preview)
- Module: Transformers

## A glimpse of Self Attention from Scratch

Self-attention lets each token combine information from other tokens. It compares a query with keys, turns the scores into weights, then mixes the value vectors using those weights.

The output is a weighted mixture, not a copy of the token with the highest score. Changing one weight changes how much that token contributes to the mixture.

## Continue learning

Pro unlocks guided learning, complete notes, practical walkthroughs, exercises, and every AI Research lesson.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md