Transformer Architecture Refresher

The tensors you can now hold in code implement one architecture, repeated. Llama 3.1 8B is a transformer: an embedding matrix, a stack of 32 identical blocks, and an output matrix, with every shape in the stack already sitting in the inventory from tensor shapes. This lesson is the refresher that assembles those parts into a map.

A map is all it is. What each part computes when a real prompt flows through, the numbers themselves, is covered in understanding one token, one lesson per part. Here the job is to name the parts, pin their shapes, and leave you able to read an architecture diagram the way the lesson on tensor shapes taught you to read a config file.

The skeleton

A token enters as its row of the embedding matrix, the [128256, 4096] table from tensor shapes: one 4,096-wide vector per vocabulary entry. From that point on, the model maintains one tensor of shape [seq, 4096], one row per token of the input. Call it the residual stream; it is the spine everything else hangs off.

Each of the 32 blocks reads the stream, computes something from it, and adds the result back into the stream. The shape never changes: [seq, 4096] in, [seq, 4096] out, 32 times. After the last block, a final normalization and the output matrix (the second [128256, 4096] giant from the inventory) map each row back over the vocabulary: one score per entry, the scores that sampling and statistics turned into a distribution.

Inside one block

A block is two sublayers in sequence, each wrapped the same way: normalize the stream, run the sublayer, add its output back to the stream. The wrapper is why the architecture trains and runs at depth 32 without falling apart, and it is why the stream is called residual: sublayers compute adjustments, the stream accumulates them.

The normalization is RMSNorm, a rescaling of each row using the [4096] scale vectors the lesson on tensor shapes counted 65 of: one before each sublayer in each block (64), plus the final one. The config's rms_norm_eps field is this layer's fingerprint.

Attention, structurally

The first sublayer is attention, and you already know its equation from attention math. What the refresher adds is where Q, K, and V come from: the block multiplies the normed stream by three weight matrices of its own. In the weights file they sit as q_proj [4096, 4096], and as k_proj and v_proj, both [1024, 4096], and their outputs are Q, K, and V. What the letters mean during generation stays with understanding one token.

The sublayer is multi-head: the 4,096-wide query vectors are split into 32 heads of width 128 (the d = 128 that the lesson on attention math ran the equation at), and each head runs the equation independently on its slice. The heads' outputs are concatenated back to 4,096 wide and passed through one more matrix, o_proj [4096, 4096], to re-enter the stream.

The quarter-width K and V matrices are the config's num_key_value_heads: 8 made visible: 1024 = 8 × 128. Only 8 K and V heads exist, and the 32 query heads share them in groups of 4. The arrangement is called grouped-query attention, and it is why the weights file shows K and V projections a quarter the size of Q. What that sharing buys at serving time is covered in prefill, decode, and KV memory.

Common misconception. Attention is the transformer's famous part, so engineers assume it is most of the model. Count instead: per block, the four attention matrices hold 41,943,040 parameters while the MLP trio holds 176,160,768, four fifths of the block.

The gated MLP

The second sublayer is the MLP, and its three matrices are the widest in the model. The up projection [4096 → 14336] and the gate projection (the same shape, the twin the lesson on tensor shapes left unnamed) both widen the row in parallel. The gate's output passes through SiLU, the smooth switch the config names in hidden_act, and multiplies the up projection's output elementwise: a learned filter deciding, position by position, how much of the widened signal passes. The down projection [14336 → 4096] returns the result to stream width. The trio is called SwiGLU in the literature.

Position

What remains is position. The attention equation treats its rows as a set: QKᵀ compares every row against every row, and nothing in the arithmetic says which token came first. Order enters through rotary position embeddings, RoPE: before the dot products, each Q and K row is rotated by angles that depend on its position, so the scores become position-aware. The config carries its settings (rope_theta, and a rope_scaling block that stretches the scheme to the model's 131,072-token maximum context). The fact to keep is that position lives inside attention, not in the embedding matrix.

The map is complete: embed, then 32 rounds of attend-and-adjust, normalize throughout, score at the end. Every matrix in the weights file now has a job, and every job runs on the arithmetic from attention math. Before understanding one token can trace a prompt through it, you need somewhere for the machine to run: the next lesson gets you onto a Linux box with a GPU.