SwiGLU: two projections and a gate
SwiGLU multiplies a SiLU-activated projection by a second projection, then projects back. Follow every coordinate through a tiny PyTorch block.
A feed-forward block can make two different views of the same input, multiply them coordinate by coordinate, then turn the result back into an output vector. One view controls how strongly values from the other pass through.
SwiGLU uses this structure with a smooth function called SiLU on the controlling branch. The useful starting point is the multiplication. The word “gate” does not mean this branch contains probabilities.
Two views of one input
For a bias-free block, use an input row x with d features and a hidden width h:
| Step | Calculation | Last dimension |
|---|---|---|
| Controlling branch | SiLU(xW_gate) | h |
| Value branch | xW_up | h |
| Combine | Controlling values × value-branch values, elementwise | h |
| Project back | Combined row × W_down | d |
The two input matrices have shape d × h; the output matrix has shape h × d. Their values are learned separately. This differs from simply applying a new activation to every original input coordinate.
SiLU, also called Swish with parameter 1, is:
$$\operatorname{SiLU}(a)=a\sigma(a),\qquad\sigma(a)=\frac{1}{1+e^{-a}}$$
At a = 1, the sigmoid is about 0.731, so SiLU is about 0.731. If the matching value-branch coordinate is 2, their product is about 1.462.
What happens to that product if the controlling input is −1 instead?
Inspect a complete tiny block
This PyTorch program uses chosen matrices so you can follow every coordinate. Install PyTorch using its platform instructions if needed; the example runs on CPU.
The third product coordinate contributes to both output coordinates because the last row of W_down is [1, 1]. Follow that same path for a change: set the last entry of the first row of w_up from 1 to 0. Predict the product and output, then rerun.
What the comparison needs to control
This toy checks the mechanism, not training quality. A gated block has three projection matrices instead of the two in a simple ungated feed-forward block. At the same hidden width, their parameter counts differ.
In GLU Variants Improve Transformer, read equations 5–6, then the paragraph that reduces hidden width to two-thirds for the gated comparison. Table 1 reports results for that study's settings. Those results support trying the block; they do not guarantee it beats ReLU or GELU on every task.
Read real code: In this pinned SwiGLU implementation, find the split into gate and up, their multiplication, and the output projection. A combined input projection computes both branches at once; the two resulting halves still have the distinct jobs shown above.
- A bias-free SwiGLU block computes two projections, applies SiLU to one, multiplies coordinate by coordinate, then projects back.
- SiLU(a) = a × sigmoid(a). Its output can be negative or greater than one; the gate is not a probability.
- Input matrices have shape d × h, and the output matrix h × d. Zeroing a value-branch coordinate zeroes that product coordinate.
- Three projection matrices change the parameter budget. The original study reduced gated hidden width to make its comparison fairer.
- A tiny numerical block verifies the operation. Better training results require an actual controlled experiment.