Self-Study LLM

LLM From Scratch

A glimpse of Self-Study LLM

A model can own several small feed-forward networks and choose which ones process each token. Those networks are called experts. A router scores the experts for the current token, and a sparse mixture-of-experts block runs only the selected ones.

The word “expert” does not mean we assigned one network to mathematics and another to grammar. Their behavior is learned from training.

Continue learning

Pro unlocks guided learning, complete notes, practical walkthroughs, exercises, and every AI Research lesson.