Softmax temperature explained with numbers

Softmax temperature explained in one line: divide every logit by the positive number T before applying softmax.

That one division sharpens or flattens a probability distribution. It does not have one universal purpose, though. Sampling, calibration, training, and distillation use the same equation for different jobs.

The distinction matters more than memorizing that low temperature is "confident" and high temperature is "creative."

Softmax temperature explained by logit gaps

For logit z i, the temperature-scaled probability is exp(z i / T) divided by the sum of exp(z j / T) over all choices j.

The knowledge-distillation paper writes this equation with T normally set to 1. A value above 1 produces a softer distribution.

The easiest way to read the formula is through a pair of choices. Their probability ratio is exp((z a minus z b) / T).

The normalization sum cancels from that ratio. Temperature therefore divides every logit gap before the exponential turns the gap into odds.

If the gap is 1, the higher-logit choice is exp(1), or about 2.718, times as likely at T=1.

At T=0.5, the effective gap is 2, so the ratio becomes exp(2), or about 7.389. At T=2, the effective gap is 0.5 and the ratio falls to about 1.649.

Temperature does not add random noise to logits. It changes how strongly all existing gaps affect the normalized probabilities.

Work the same logits at T=0.5, 1, and 2

Take three logits: 2, 1, and 0. The first choice ranks highest, but logits are scores rather than probabilities.

At T=1, the scaled logits stay 2, 1, and 0. Their exponentials are about 7.389, 2.718, and 1.000, with a sum of 11.107.

The probabilities are therefore about 0.665, 0.245, and 0.090.

At T=0.5, divide by one half. The scaled logits become 4, 2, and 0, with exponentials about 54.598, 7.389, and 1.000.

After division by their sum, the probabilities are about 0.867, 0.117, and 0.016. The leading choice now holds nearly seven eighths of the mass.

At T=2, the scaled logits become 1, 0.5, and 0. Their exponentials are about 2.718, 1.649, and 1.000.

The probabilities are about 0.506, 0.307, and 0.186. The first choice still leads, but the tail receives much more mass.

Fanout's softmax and cross-entropy workbench shows how logits become probabilities before a target loss is computed.

The three calculations above keep the logits fixed. Any behavioral difference comes from changing T, not from asking the model to recompute its scores.

Temperature preserves the winner

For any positive finite T, dividing two logits by the same positive number preserves their order.

Exponentiation also preserves order, and the shared softmax denominator cannot reverse it. The largest logit remains the largest probability.

This is why temperature alone cannot change a greedy argmax choice. It changes the distribution from which a sampler may draw.

The calibration study by Guo and colleagues makes the same point for classifiers. A single positive temperature changes confidence but not the predicted class or accuracy.

As T approaches zero from above, the distribution approaches a point mass on the largest logit. As T grows without bound, all finite scaled logits approach zero and softmax approaches a uniform distribution.

T=0 itself is not a valid substitution because division by zero is undefined. An API may interpret zero as a request for greedy decoding, but that is an implementation convention outside the formula.

Fanout's argmax versus max guide is useful here: temperature changes probability values while preserving the returned argmax for positive T.

Decoding temperature only matters when sampling

During autoregressive generation, a model produces a new logit vector at every token position. A decoder can scale that vector, normalize it, and sample one next token.

The Hugging Face generation contract separates `temperature` from `do_sample`. With sampling disabled, generation uses a greedy strategy instead.

That makes the arithmetic concrete. For logits 2, 1, and 0, greedy decoding picks the first choice at T=0.5, 1, or 2 because its rank never changes.

Sampling behaves differently. Across many draws, the first choice should appear about 87 percent of the time at T=0.5 and about 51 percent at T=2, before any top-k or top-p filter changes the candidate set.

One completion is weak evidence about temperature. A low-temperature draw can still select a tail token, and a high-temperature draw can still select the top token.

Temperature is also not a factuality control. It reshapes probabilities supplied by the model; it does not verify claims or repair missing knowledge.

Top-k and top-p do a different job. They remove candidates before renormalization, while temperature changes relative odds among the candidates that remain.

Calibration chooses T from held-out labels

A classifier can choose the correct class often while reporting probabilities that are too high or too low.

Calibration asks whether predictions made with about 80 percent confidence are correct about 80 percent of the time over a suitable group.

Guo and colleagues fit one positive scalar T on a held-out validation set by minimizing negative log likelihood. The network weights stay fixed.

Because the same scalar divides every class logit, the predicted label remains unchanged. The fitted probabilities can become less overconfident without improving classification accuracy.

This is not the same as choosing a decoding temperature for writing style. Calibration has labeled validation data and a scoring objective. Generation temperature is usually a user or system setting applied at each token.

The fitted calibration temperature is retained when calibrated confidence estimates are needed on later data drawn from the same distribution.

That last assumption is easy to omit. A T fitted on one validation distribution need not repair confidence after a large distribution shift.

Distillation uses temperature to expose similarities

Knowledge distillation gives a student model targets produced by a larger teacher.

At T=1, a confident teacher may assign nearly all probability to the correct class. Tiny non-target probabilities still contain information about which mistakes the teacher considers similar.

The Hinton, Vinyals, and Dean paper raises the teacher's softmax temperature to reveal those relative non-target probabilities. The student uses the same high temperature while matching the soft targets.

This is a training signal, not a request for more varied output. The paper returns the student to T=1 after training.

It also notes that soft-target gradient magnitudes scale as 1 divided by T squared. When hard- and soft-target losses are mixed, the soft term is multiplied by T squared to keep their relative influence comparable.

The numerical effect is familiar. With logits 2, 1, and 0, T=2 gives the student meaningful target mass of about 0.307 and 0.186 on the two alternatives.

Those alternatives would receive only about 0.117 and 0.016 at T=0.5. A sharper teacher hides more of its class-similarity structure.

Training temperature changes the optimization problem

Temperature can also appear inside the softmax used while training the original model.

Now T does more than post-process a fixed model. It changes probabilities, losses, and gradients used to update the weights.

A 2025 study of temperature-scaled classification analyzes this distinction and reports that temperature affects both gradient scale and optimization direction in its training setup.

That result should not be collapsed into the decoding slogan that higher T means more randomness. No token sampling is required for a training loss to change.

When reading a paper, locate T in the computation graph. Ask whether logits are scaled during base training, post-training calibration, teacher-student transfer, or final decoding.

Then ask who selects T. It may be tuned against validation likelihood, set as a training hyperparameter, paired across teacher and student, or exposed to an application user.

The equation is identical. The objective, data, and lifetime of the parameter are not.

Compute temperature without numerical overflow

Directly exponentiating large scaled logits can overflow, especially when T is small.

Subtract the largest scaled logit before exponentiation. Softmax is unchanged because multiplying every numerator and the denominator by the same constant cancels out.

For logits 2, 1, and 0 at T=0.5, use shifted scaled logits 0, -2, and -4 instead of 4, 2, and 0.

Their exponentials are 1, about 0.1353, and about 0.0183. Normalizing them still gives about 0.867, 0.117, and 0.016.

The PyTorch softmax documentation also requires the normalization dimension.

For a batch-by-vocabulary tensor, the usual generation axis is vocabulary, not batch.

A production implementation may fuse scaling, filtering, softmax, and sampling. The conceptual order still matters when comparing outputs across libraries.

Read any temperature setting with four questions

First, identify the logits and the axis being normalized. A single scalar T can be shared across classes, tokens, or examples only if the implementation says so.

Second, determine whether the next operation is argmax or sampling. Positive temperature changes sampling odds but cannot change the argmax ranking.

Third, name the objective that selected T. Validation likelihood, distillation loss, training accuracy, and subjective generation quality are different targets.

Fourth, check when T returns to 1. Distillation commonly uses a high T only while transferring knowledge, while calibration commonly keeps the fitted T for later confidence estimates.

Temperature divides logit gaps. Its effect on the system depends on where that division is applied.