SwiGLU: two projections and a gate

SwiGLU multiplies a SiLU-activated projection by a second projection, then projects back. Follow every coordinate through a tiny PyTorch block.

A feed-forward block can make two different views of the same input, multiply them coordinate by coordinate, then turn the result back into an output vector. One view controls how strongly values from the other pass through.

SwiGLU uses this structure with a smooth function called SiLU on the controlling branch. The useful starting point is the multiplication. The word “gate” does not mean this branch contains probabilities.

Two views of one input

For a bias-free block, use an input row x with d features and a hidden width h:

Step Calculation Last dimension --- --- --- Controlling branch SiLU(xW gate) h Value branch xW up h Combine Controlling values × value-branch values, elementwise h Project back Combined row × W down d

The two input matrices have shape d × h; the output matrix has shape h × d. Their values are learned separately. This differs from simply applying a new activation to every original input coordinate.

SiLU, also called Swish with parameter 1, is:

At a = 1, the sigmoid is about 0.731, so SiLU is about 0.731. If the matching value-branch coordinate is 2, their product is about 1.462.

What happens to that product if the controlling input is −1 instead?

Inspect a complete tiny block

This PyTorch program uses chosen matrices so you can follow every coordinate. Install PyTorch using its platform instructions if needed; the example runs on CPU.

python title="swiglu block.py" import torch from torch.nn import functional as F