---
title: "LayerNorm and RMSNorm"
description: "Shows exactly which centering and bias operations RMSNorm removes from LayerNorm while retaining feature-wise rescaling."
canonical_url: "https://fanout.sh/labs/math-decoder/formula/layernorm-versus-rmsnorm"
md_url: "https://fanout.sh/labs/math-decoder/formula/layernorm-versus-rmsnorm.md"
last_updated: "2026-08-09"
access: "public"
---

# LayerNorm and RMSNorm

Shows exactly which centering and bias operations RMSNorm removes from LayerNorm while retaining feature-wise rescaling.

## Public overview

LayerNorm centers x by its mean and scales by standard deviation before gain and bias, while RMSNorm skips centering and scales x by its root mean square before gain.

LayerNorm removes feature offset and scale, while RMSNorm removes only scale and performs fewer statistics.

Normalization by a feature-group scale is common to both; mean subtraction is the decisive structural difference.

The RMSNorm paper argues that re-scaling invariance can be retained without LayerNorm's re-centering operation.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md