Softmax and cross-entropy from logits

A classifier usually emits logits, not probabilities. They may be negative, greater than one, and shifted by an arbitrary common constant. Softmax turns their relative differences into a distribution; cross-entropy asks how much probability reached the target.

The two operations are often fused in software and compressed into one loss symbol in papers. Reading them separately first exposes the axis, normalization, numerical stability, and the exact event being penalized.

This guide follows one logit vector from scores to probabilities to loss, then tests the invariant that makes stable implementations possible.

Logits are scores on a relative scale

Let z contain one real-valued score per class. A larger score means the model favors that class relative to classes in the same normalized group. A logit by itself is not a calibrated probability.

The class axis is part of the equation. For a batch by classes tensor, softmax usually normalizes across classes for each example. Normalizing across the batch would couple unrelated examples and answer a different question.

Before calculating, write the contract: z has C entries, y identifies a target class or target distribution, p will have C entries, and the unreduced one-example loss will be a scalar.

Softmax converts differences into a distribution

For class i, p i equals exp of z i divided by the sum over j of exp of z j. Exponentials make every numerator positive, and dividing by their shared sum makes all p values add to one.

Softmax preserves order: if z a is greater than z b, then p a is greater than p b. It does not preserve numerical gaps, because every probability also depends on all other logits in the group.

PyTorch's Softmax documentation specifies that outputs lie between zero and one and sum to one along the chosen dimension. It also directs log-probability uses toward LogSoftmax for better numerical properties.

Cross-entropy measures the target surprise

For a single hard target class y, categorical cross-entropy is L equals negative log p y. If the model gives the target high probability, the loss is small. If target probability approaches zero, the loss grows without bound.

With a target distribution y i, the expression becomes negative sum over i of y i log p i. A one-hot target reduces that sum to the single target term because every non-target weight is zero.

Cross-entropy is not classification accuracy. Two predictions can choose the same top class while assigning different probability to it, producing the same accuracy and different losses.

PyTorch's CrossEntropyLoss documentation accepts unnormalized logits and states that the class-index form is equivalent to LogSoftmax followed by negative log likelihood. That API contract is why applying softmax first is usually unnecessary.

Combine the steps with log-sum-exp

Substitute softmax into negative log p y. The one-label loss becomes log of the sum over j of exp z j, minus z y. The first term summarizes competition from every class; the second rewards the target score.

Raising only the target logit lowers the loss. Raising a non-target logit raises the shared log-sum-exp term and therefore raises the target loss. The objective couples all classes even though the compact hard-label formula names only y.

The formula also explains gradients: the derivative with respect to each logit is p i minus y i for the standard one-example categorical case. The target is pushed up when underweighted, while non-targets are pushed down in proportion to their probability.

Subtract the maximum without changing the answer

Exponentials can overflow for large positive logits. Let m be the maximum logit and compute exp of z i minus m over the sum of exp of z j minus m.

This is exactly equal in real arithmetic because the common factor exp of negative m cancels between numerator and denominator. It is not an approximation or a different temperature.

After the shift, the largest exponent is exp zero, which equals one, and every other exponent is at most one. The same idea produces the stable loss logsumexp of z minus z y without materializing vulnerable raw exponentials.

TensorFlow's softmax cross-entropy with logits documentation likewise expects logits and performs the combined operation. Framework contracts matter because passing probabilities where logits are expected changes the computation.

Shift every logit and watch nothing change

The workbench lets you choose a target class and move its logit. Then add 1,000 to every score. Stable probabilities and cross-entropy remain the same, while a naive direct exponential calculation overflows.

This is a stronger test than checking one memorized output. It verifies softmax's shift invariance and shows why implementation form matters even when the mathematical function is unchanged.

Use the categorical cross-entropy formula case for the full symbol ledger, or review the softmax notation entry before changing the target.

Check invariants and extreme cases

Adding one constant to every logit changes neither probabilities nor loss. Permuting logits permutes probabilities in the same way. Softmax outputs remain positive and sum to one under exact real arithmetic.

If all C logits are equal, every class receives probability one over C. If one logit exceeds the others by an increasingly large margin, its probability approaches one and the others approach zero.

For a hard target, p y approaching one sends loss toward zero. Sending p y toward zero sends loss upward without a finite ceiling. Finite-precision implementations may reach representational limits before those mathematical extremes.

Name the target and normalization assumptions

The one-label explanation assumes mutually exclusive classes normalized along one class axis. Multi-label problems usually need independent sigmoid outputs and a binary cross-entropy-style objective instead of one categorical softmax.

Soft targets, label smoothing, class weights, ignored indices, and reduction across batches modify the objective. Papers may average per token, per sequence, or per example, so inspect the summation and denominator.

Logit temperature changes relative gaps by dividing scores before softmax. A common additive shift leaves gaps unchanged. Confusing those operations leads to the false belief that numerical stabilization changes model confidence.

Loss does not tell the whole prediction story

Low cross-entropy on an evaluation set does not guarantee calibration, fairness, robustness, or good behavior under distribution shift. It measures a particular probabilistic scoring objective under the supplied labels and reduction.

Very large vocabularies and sampled or approximate objectives can change how normalization is computed during training. Distributed sharding can split the class axis while preserving the mathematical reduction through communication.

The workbench uses three classes and ordinary double-precision browser arithmetic. It illustrates semantics and stability, not the fused-kernel performance or rounding behavior of a production training system.

Ask a mechanism-level question

If the same constant is added to every logit, do target probability and one-label cross-entropy change.

Neither changes. Softmax depends on relative differences, and the common exponential factor cancels. A naive computation may still overflow, which is a numerical failure rather than a change in the mathematical answer.

Frequently asked questions

Should I apply softmax before CrossEntropyLoss

Not for the standard PyTorch class described above. It expects raw logits and combines log-softmax with negative log likelihood. Always verify the exact library API instead of transferring this rule by name alone.

Can a logit be negative

Yes. Logits are unconstrained real scores. Their shared offset is arbitrary under softmax; relative differences determine the probabilities.

Why use logarithms in the loss

Negative logarithm assigns a small penalty near target probability one and an increasingly large penalty as target probability approaches zero. It also converts products of likelihoods into sums across independent observations.

Carry the axis and target forward

Continue with the scaled attention guide to see softmax normalize routing scores, or practice type, scope, and shape reading in the Math Decoder curriculum.

The dependable reading is short: logits are relative scores, softmax normalizes one declared axis, cross-entropy reads the target probability, and stable log-sum-exp computes the same function without dangerous raw exponentials.