Root Mean Square Layer Normalization

RMSNorm asks whether LayerNorm needs to subtract the mean as well as control activation scale. Read it to see how dividing by a vector's root mean square preserves global rescaling invariance with less normalization work, while deliberately giving up re-centering invariance.

Reading focus: How LayerNorm's centering and scaling operations can be separated into distinct invariance properties. Why dividing an activation vector by its RMS makes positive global rescaling cancel while leaving additive shifts visible. Why the paper's quality and runtime results support a useful simplification without proving that mean-centering is universally dispensable.

NeurIPS 2019. Zhang and Sennrich. 35 min read, medium difficulty.