RMSNorm

Neural Network from Scratch

A glimpse of RMSNorm

A network layer may receive vectors whose overall magnitude varies. RMSNorm rescales a vector using the root mean square of its entries, then applies a learned scale to each feature.

Start with [3, 4]. Square the entries to get [9, 16], average them to get 12.5, then take the square root: approximately 3.536. Dividing both entries by that number gives approximately [0.849, 1.131].

Continue learning

Pro unlocks guided learning, complete notes, practical walkthroughs, exercises, and every AI Research lesson.