---
title: "Toy Models of Superposition"
description: "Toy Models of Superposition asks how a neural layer can track more useful features than it has hidden dimensions. Read it to see how sparse features share non-orthogonal directions, why ReLU makes interference affordable, and why the clean geometry remains a controlled clue rather than a map of a real language model."
canonical_url: "https://fanout.sh/daily/2026-10-10-toy-models-superposition"
md_url: "https://fanout.sh/daily/2026-10-10-toy-models-superposition.md"
last_updated: "2026-10-10"
access: "public"
---

# Toy Models of Superposition

Toy Models of Superposition asks how a neural layer can track more useful features than it has hidden dimensions. Read it to see how sparse features share non-orthogonal directions, why ReLU makes interference affordable, and why the clean geometry remains a controlled clue rather than a map of a real language model.

## Paper details

- Authors: Nelson Elhage et al.

- Venue: Transformer Circuits Thread 2022

- Track: AI Research

- Difficulty: Medium

- Reading time: 34 min

- Original paper: https://arxiv.org/abs/2209.10652v1

## What you will learn

- How a tied-weight ReLU autoencoder turns features into directions through a narrow hidden layer.

- Why sparse features can share non-orthogonal directions while dense features are pruned.

- How phase changes and geometric packing sharpen interpretability questions without proving what happens in real models.

## Continue exploring

[Map the interpretability stack](https://fanout.sh/knowledge-graph?node=ai-research%3Aconcept%3Ainterpretability): Connect the toy mechanism to feature discovery, sparse representations, and causal tests in the Fanout knowledge map.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md