Label smoothing with sparse outputs

Label smoothing with sparse outputs has a boundary that ordinary softmax examples hide. If a smoothed target assigns positive mass to every class but the prediction assigns zero to one class, the cross-entropy loss is infinite.

An entropy confidence penalty does not have that failure. Prediction entropy stays between zero and log K, even when the prediction contains exact zeros.

The difference comes from the direction of the KL divergence, not from two interchangeable ways to discourage confidence.

Label smoothing with sparse outputs

Szegedy and colleagues define label smoothing by mixing a hard target with a prior distribution.

For K classes, let q be the one-hot target, u the uniform distribution, and epsilon the smoothing weight. The smoothed target is q prime = (1 - epsilon)q + epsilon u.

The training loss is H(q prime, p), where p is the model prediction. It splits into (1 - epsilon)H(q, p) + epsilon H(u, p).

Uniform smoothing gives every class positive target mass. Under the common all-class convention, even an incorrect class receives epsilon divided by K.

That is harmless when p comes from finite softmax logits because every softmax probability is strictly positive. It becomes important when the output transform, pruning step, or stored distribution can contain exact zeros.

Fanout's label smoothing minimum-loss guide derives the positive loss floor under dense softmax predictions. Sparse predictions expose the other boundary.

The KL direction determines the boundary

The uniform cross entropy can be written as H(u, p) = log K + KL(u || p). The prediction sits in the second argument of the KL divergence.

If u_j is positive and p_j is zero, the j term is u_j times log(u_j / p_j). Division by zero sends that term to infinity.

The confidence penalty from Pereyra and colleagues instead rewards prediction entropy H(p).

Prediction entropy can be written as log K - KL(p || u). Here p is in the first argument.

When p_j is zero, its entropy contribution is defined by the limit of negative p_j log p_j. That limit is zero, so the term remains finite.

The full entropy lies between zero and log K. A point mass has entropy zero, while the uniform distribution reaches log K.

The label smoothing versus confidence penalty guide compares these objectives. Sparse outputs make their different KL directions impossible to ignore.

A four-class calculation

Take four classes, true class 1, and epsilon = 0.1. Under all-class smoothing, the target is [0.925, 0.025, 0.025, 0.025].

Now take the sparse prediction p = [0.9, 0.1, 0, 0]. Its hard-label negative log likelihood is negative log 0.9, about 0.105 nats.

The smoothed loss also contains negative 0.025 log 0 for class 3 and the same term for class 4. Either term is infinite, so the finite target-class score cannot rescue the total.

Replace each zero with 0.000001 and make the tiny normalization adjustment. The two floor terms contribute about 0.691 nats together.

The full smoothed loss is about 0.846 nats. Lowering the floor by another factor of ten adds about 0.115 nats because two classes each carry target mass 0.025.

The same sparse prediction has entropy near 0.325 nats. With confidence-penalty weight beta = 0.1, the objective NLL - beta H(p) is about 0.073.

That number is finite without a probability floor. It is not equivalent to the smoothed loss; it measures a different departure from uniformity.

Some libraries distribute epsilon only across the K - 1 incorrect classes. That convention changes the coefficients and the worked number, but every incorrect class still has positive target mass.

An exact zero in any of those classes still makes the smoothed cross entropy infinite.

Near-zero probabilities still matter

Finite softmax logits avoid exact zeros in real arithmetic, yet floating-point underflow and explicit sparsification can still produce them in an implementation.

Even before zero, the label-smoothing term grows without bound as p_j approaches zero. The probability-domain derivative of negative q_j log p_j is negative q_j divided by p_j.

That derivative can become large for a tiny p_j. The final parameter gradient also depends on the output transform, so do not copy the probability derivative as a logit-gradient formula.

Dense softmax cross entropy is usually evaluated from logits with log-sum-exp. That stable path avoids forming a rounded probability and then taking its logarithm.

If a pipeline converts logits to probabilities, applies top-k pruning, and feeds the pruned vector into a smoothed cross entropy, it has changed both the numerical path and the objective's support.

The safe question is not only whether the vector still sums to one. Ask whether every class with positive target mass also has positive predicted mass.

Exact zeros can be intentional

Sparsemax was designed to return sparse probability distributions, including exact zeros. Other sparse transforms make the same support choice for interpretability or structured selection.

There is no numerical bug in an exact zero produced by such a transform. The conflict appears when a full-support smoothed target is paired with that sparse prediction under cross entropy.

This is a support mismatch. The target says every class remains possible, while the prediction says some classes are impossible.

Label smoothing is doing what its definition requires when it assigns infinite loss to that pair. Whether that behavior is useful depends on why the prediction was made sparse.

If sparsity is only a serving optimization, train against dense logits and prune after the loss. Training does not need to inherit a deployment representation.

If sparsity is part of the model, choose a loss whose domain includes the intended boundary or document the approximation used to move away from zero.

Three defensible implementation choices

First, keep a dense logit path for training. Use the framework's logits-based cross entropy with label smoothing, then apply top-k or thresholding only to the inference output.

This preserves the full-support softmax model assumed by standard label smoothing. It also keeps log-sum-exp inside the fused loss instead of logging rounded probabilities.

Second, apply a probability floor before the logarithm. Replace each p_j with max(p_j, delta), then renormalize if the loss expects a distribution.

This caps the contribution from one smoothed class near negative q_j log delta. It prevents infinity, but delta becomes part of the objective and should be recorded as a hyperparameter.

Third, use a confidence penalty when the actual goal is to discourage low-entropy predictions without imposing positive target mass on every class.

The confidence penalty stays bounded at sparse boundaries. It can still change optimization and calibration, so bounded does not mean automatically better.

Muller, Kornblith, and Hinton also show that label smoothing can erase information useful for distillation.

Regularizer choice should follow the information and support constraints of the task, not a generic preference for less confident outputs.

Test the boundary directly

Create one test prediction with ordinary dense probabilities, one with probabilities near the smallest normal value of the training dtype, and one with exact zeros.

Run the loss under float32 and the mixed-precision dtype used in training. Assert whether the intended result is finite, infinite, or rejected with a clear error.

Then compare the logits-based implementation with any probability-based path. A finite fused loss and an infinite post-pruning loss reveal where support was lost.

Check reduction behavior too. One infinite sample can make a batch mean infinite, while an ignored or masked sample may hide the issue until a different batch composition appears.

Log the minimum predicted probability before the loss and after every transform. A single summary taken only before pruning cannot diagnose a zero introduced later.

For a floor, sweep delta by powers of ten. The loss should rise linearly with negative log delta, with a slope equal to the total smoothed target mass placed on floored classes.

That slope is a useful correctness check. It distinguishes the expected reverse-KL penalty from an unrelated overflow or invalid normalization.

Choose the constraint you mean

Label smoothing says the target distribution has full support. Its reverse KL term strongly rejects a prediction that removes any class from that support.

A confidence penalty says the prediction should retain entropy. Its forward KL view tolerates zero coordinates because those coordinates contribute no probability mass.

For dense softmax classifiers, both can discourage extreme confidence and both may work well. Their behavior separates near the simplex boundary.

Use label smoothing when a positive prior over every class is part of the training assumption. Keep the loss on logits so the model prediction retains full support.

Use a probability floor only when its bias is acceptable and measured. Use a confidence penalty when bounded output entropy is closer to the intended constraint.

Before tuning the regularizer, list which target coordinates can be positive. Then verify that the prediction is positive on those same coordinates.