---
title: "Llama 4 From Scratch"
description: "Build Llama 4's MoE feed-forward block: a router, a sigmoid gate on the expert's input, a SwiGLU expert, and the shared expert every token uses."
canonical_url: "https://fanout.sh/ai/lessons/llm-from-scratch-llama-4"
md_url: "https://fanout.sh/ai/lessons/llm-from-scratch-llama-4.md"
access: "public"
---

# Llama 4 From Scratch

Build Llama 4's MoE feed-forward block: a router, a sigmoid gate on the expert's input, a SwiGLU expert, and the shared expert every token uses.

## Public overview

A large model can store many feed-forward networks without running all of them for every token. Llama 4 uses this idea: a router selects a network, called an expert, for the token being processed.

There is also a shared expert that processes every token. That gives us a small piece of the architecture we can build and inspect: one shared path plus one selected path.

Follow one token through an expert block

Suppose a token is represented by the vector x = [1, 0]. A learned router produces a score for each available expert. We will use two experts and scores [0, −1], so expert 0 wins.

Llama 4 applies sigmoid to the winning score. Sigmoid maps a number into the range 0 to 1; sigmoid(0) is 0.5. The selected expert receives 0.5x , while the shared expert receives x .

Branch Input to expert Contribution to the result --- --- --- Shared x = [1, 0] Shared expert's output Routed 0.5x = [0.5, 0] Expert 0's output on the scaled input Combination Both branch outputs Add the two vectors

Writing S for the shared network and E for the selected network:

Here r j is the winning router score. This y is the feed-forward branch's output; a full decoder block also has normalization, attention and residual additions.

It is tempting to write the second term as sE j(x). Would that give the same answer? For a linear expert, yes. For this SwiGLU expert, generally no: the position of that multiplication matters.

Build the nonlinear part

A Llama 4 expert uses SwiGLU : make two projections of the input, apply SiLU to one, multiply their entries, then project back. In PyTorch notation that is down(silu(gate(x)) up(x)) .

SiLU(z) = z × sigmoid(z), applied separately to each coordinate.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md