---
title: "Root Mean Square Layer Normalization"
description: "RMSNorm asks whether LayerNorm needs to subtract the mean as well as control activation scale. Read it to see how dividing by a vector's root mean square preserves global rescaling invariance with less normalization work, while deliberately giving up re-centering invariance."
canonical_url: "https://fanout.sh/daily/2026-08-06-rmsnorm"
md_url: "https://fanout.sh/daily/2026-08-06-rmsnorm.md"
last_updated: "2026-08-06"
access: "public"
---

# Root Mean Square Layer Normalization

RMSNorm asks whether LayerNorm needs to subtract the mean as well as control activation scale. Read it to see how dividing by a vector's root mean square preserves global rescaling invariance with less normalization work, while deliberately giving up re-centering invariance.

## Paper details

- Authors: Zhang and Sennrich

- Venue: NeurIPS 2019

- Track: AI Research

- Difficulty: Medium

- Reading time: 35 min

- Original paper: https://papers.nips.cc/paper_files/paper/2019/hash/1e8a19426224ca89e83cef47f1e7f53b-Abstract.html

## What you will learn

- How LayerNorm's centering and scaling operations can be separated into distinct invariance properties.

- Why dividing an activation vector by its RMS makes positive global rescaling cancel while leaving additive shifts visible.

- Why the paper's quality and runtime results support a useful simplification without proving that mean-centering is universally dispensable.

## Continue exploring

[Explore RMSNorm](https://fanout.sh/knowledge-graph?node=ai-research%3Aconcept%3Armsnorm): Open Fanout's RMSNorm node and connect the paper to LayerNorm, activation scale, transformer blocks, and the neural-network-from-scratch course.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md