Self Attention from Scratch

Transformers

A glimpse of Self Attention from Scratch

Self-attention lets each token combine information from other tokens. It compares a query with keys, turns the scores into weights, then mixes the value vectors using those weights.

The output is a weighted mixture, not a copy of the token with the highest score. Changing one weight changes how much that token contributes to the mixture.

Continue learning

Pro unlocks guided learning, complete notes, practical walkthroughs, exercises, and every AI Research lesson.