The scaled dot-product attention equation

The attention equation compresses four different operations into one line: compare queries with keys, control the scale of those comparisons, normalize them into weights, and use the weights to mix values. Reading it as one mysterious function hides the mechanism.

A useful explanation therefore starts before the formula. One token asks a query. Every token exposes a key used for matching and a value containing information to retrieve. Attention computes how much of each value should contribute to the query's new representation.

The square-root term is especially easy to misread. It does not make the matrix smaller and it does not make weights sum to one. It controls the typical magnitude of dot-product logits before softmax when the key dimension grows.

Begin with the retrieval question

Suppose the token robot needs context. Its query vector describes what information it is looking for. Each key vector describes what a position can be matched by. A dot product gives a larger score when the query and key align in their learned coordinate system.

Those scores are not yet probabilities and they do not carry the content returned to the query. They are routing logits. Softmax turns a row of logits into positive weights that sum to one, then the weighted combination of value vectors carries information forward.

This separation between address and content is an intuition, not a literal database contract. Query, key, and value projections are learned jointly. Their meaning emerges from training rather than being assigned by a human-readable schema.

The original Attention Is All You Need paper defines scaled dot-product attention and describes queries, keys, values, masking, and multi-head attention. It is the primary source for the mechanism decoded here.

Shapes reveal every pairwise comparison

For one sequence of L tokens, let Q have shape L by d k and K have shape L by d k. Transposing K gives d k by L. The product QK transpose therefore has shape L by L.

Each row belongs to one query position and each column belongs to one key position. Cell i,j is the dot-product score between query i and key j. The matrix is square because this self-attention example uses the same L positions on both sides.

Cross-attention need not be square. If target queries have length T and source keys have length S, the score matrix has shape T by S. Shape checking recovers this structure without requiring numerical multiplication.

Batch and head axes are often omitted in the paper equation. Implementations commonly carry shapes such as batch by heads by length by head dimension. The same inner-dimension rule applies within each batch and head.

The square root controls logit scale

The Transformer paper gives a simple variance argument. Assume the components of a query and key are independent, with mean zero and variance one. Their dot product sums d k products, so its variance is d k under that model.

Dividing by the square root of d k returns the variance to roughly one because variance scales with the square of a multiplicative constant. This keeps the typical logit scale from growing solely because vectors have more components.

Why does scale matter to softmax. Large differences between logits produce probabilities near zero and one. In those saturated regions, small score changes can produce very small gradients. The scaling term reduces this dimension-driven effect under the stated assumptions.

The claim is not that every trained query and key has independent unit-variance components. It is a motivation for the normalization. Learned projections, normalization layers, initialization, precision, and data all affect actual score statistics.

For the probability and linear-algebra prerequisites behind this argument, the open Mathematics for Machine Learning book develops random variables, variance, vectors, matrix multiplication, and optimization in a shared notation.

Softmax turns each score row into weights

Softmax exponentiates each scaled logit and divides by the sum of exponentials in the same row. The outputs are positive and sum to one. Adding the same constant to every logit does not change the resulting weights.

Normalization happens across keys for each query. A row answers where this query should read. Applying softmax across a different axis would answer a different question and change the mechanism even if the symbols looked similar.

Masks modify which comparisons are allowed before softmax. In causal attention, future positions receive a value that makes their softmax weight effectively zero. Padding masks prevent nonexistent tokens from collecting probability mass.

The Stanford CS229 linear algebra review provides an authoritative refresher on matrix dimensions, multiplication, transpose, and vector operations. Those contracts are enough to reconstruct the score matrix's shape.

Values carry the information that gets mixed

After softmax, the L by L weight matrix multiplies V. If V has shape L by d v, the output has shape L by d v. Each output row is a weighted sum of the value rows.

This final multiplication is why attention is more than a similarity heatmap. Scores determine routing, but values determine what content arrives. Changing V can change the output even when Q, K, and every attention weight remain fixed.

Multi-head attention repeats this process in several learned subspaces, concatenates the head outputs, and projects them again. The compact single-head formula is the mechanism's core, not a complete diagram of a Transformer block.

Follow one row through the workbench

The matrix below holds three tokens and exposes one query row at a time. Change the query, increase d k, and toggle square-root scaling. Predict whether the softmax weights become more concentrated before reading the percentages.

The toy model makes raw dot products grow with the square root of dimension. With scaling enabled, the displayed logits stay comparable. Without it, larger dimensions push the same underlying pattern toward a sharper softmax distribution.

This is a mechanism illustration, not a trained attention head. The token labels do not imply linguistically correct weights. Its purpose is to isolate the relationship among dimension, logit magnitude, softmax concentration, and matrix shape.

Use the softmax notation entry to review normalization, then return to the workbench and explain why scaling and softmax perform different jobs.

What this one-line equation leaves out

The formula does not show how X is projected into Q, K, and V. It omits batches, heads, masks, dropout, positional information, residual connections, normalization, and the output projection. Papers may also change the attention kernel or normalization.

Attention weights are not automatically faithful explanations of a model's decision. They describe a routing computation inside one layer and head. Later layers, residual paths, value content, and nonlinear transformations also influence the final output.

The workbench uses ordinary real-number arithmetic and a stable softmax calculation. Production kernels must also manage finite precision, memory traffic, masking, batching, and numerical stability. Those systems questions do not change the semantic reading of the equation.

Frequently asked questions

Why square root instead of d k

Under the paper's assumption, dot-product variance grows as d k. Dividing the dot product by square root d k divides its variance by d k, returning the scale to roughly constant order.

Does scaling make attention probabilities

No. Scaling changes the logits' magnitude. Softmax converts each row into positive normalized weights. Keeping those roles separate makes ablations and implementation details easier to read.

Is QK transpose the expensive part

Pairwise attention creates an L by L score structure, so work and memory traffic grow strongly with sequence length. Exact cost depends on implementation, head dimensions, batching, precision, and whether intermediate matrices are materialized.

Read attention as a typed program

Practice the outer-operator and shape routine in the Math Decoder curriculum, or revisit the general equation-reading guide before decoding a multi-head implementation.

The dependable reading is short: compare Q with K, scale the logits, normalize each query row, and mix V. Everything else in the equation specifies shapes, scope, and the assumptions that keep those four operations useful.