Llama 4 From Scratch

Build Llama 4's MoE feed-forward block: a router, a sigmoid gate on the expert's input, a SwiGLU expert, and the shared expert every token uses.

A large model can store many feed-forward networks without running all of them for every token. Llama 4 uses this idea: a router selects a network, called an expert, for the token being processed.

There is also a shared expert that processes every token. That gives us a small piece of the architecture we can build and inspect: one shared path plus one selected path.

Follow one token through an expert block

Suppose a token is represented by the vector x = [1, 0]. A learned router produces a score for each available expert. We will use two experts and scores [0, −1], so expert 0 wins.

Llama 4 applies sigmoid to the winning score. Sigmoid maps a number into the range 0 to 1; sigmoid(0) is 0.5. The selected expert receives 0.5x , while the shared expert receives x .

Branch Input to expert Contribution to the result --- --- --- Shared x = [1, 0] Shared expert's output Routed 0.5x = [0.5, 0] Expert 0's output on the scaled input Combination Both branch outputs Add the two vectors

Writing S for the shared network and E for the selected network:

Here r j is the winning router score. This y is the feed-forward branch's output; a full decoder block also has normalization, attention and residual additions.

It is tempting to write the second term as sE j(x). Would that give the same answer? For a linear expert, yes. For this SwiGLU expert, generally no: the position of that multiplication matters.

Build the nonlinear part

A Llama 4 expert uses SwiGLU : make two projections of the input, apply SiLU to one, multiply their entries, then project back. In PyTorch notation that is down(silu(gate(x)) up(x)) .

SiLU(z) = z × sigmoid(z), applied separately to each coordinate.