Superposition Yields Robust Neural Scaling
Strong superposition changes representation error from the cost of ignored features to interference among shared directions. Follow the controlled toy model, the one-over-width overlap geometry, the four-family LLM fit, and the causal boundary the matching exponents do not cross.
Reading focus: How decay and row growth move the same sparse autoencoder between weak and strong superposition. Why isotropic feature directions have mean squared overlap 1/m, giving interference a width exponent near one. What the OPT, GPT-2, Qwen, Pythia, and Chinchilla comparisons support, and why they do not isolate superposition as the cause of total LLM scaling.
NeurIPS 2025 Best Paper runner-up. Yizhou Liu, Ziming Liu, and Jeff Gore. 11 min read, hard difficulty.