---
title: "Label smoothing minimum loss, calculated"
description: "Derive the nonzero cross-entropy floor created by label smoothing, compare two conventions, and calculate it for 2, 10, and 1,000 classes."
canonical_url: "https://fanout.sh/blog/label-smoothing-minimum-loss"
md_url: "https://fanout.sh/blog/label-smoothing-minimum-loss.md"
last_updated: "2026-08-25"
access: "public"
---

# Label smoothing minimum loss, calculated

Derive the nonzero cross-entropy floor created by label smoothing, compare two conventions, and calculate it for 2, 10, and 1,000 classes.

- Author: Suraj Gaud

- Published: 2026-08-25

- Track: ML mathematics

- Access: Public

- Tags: label smoothing, minimum loss, cross entropy, entropy, KL divergence, PyTorch, classification, perplexity

Label smoothing minimum loss is not zero. Once the target assigns positive probability to every class, the best possible prediction matches that softened target and still has positive cross-entropy.

For 10 classes and smoothing 0.1 under PyTorch's convention, that floor is about 0.5003 nats per example. A training loss near 0.5 can be optimal for the stated target, not evidence of underfitting.

The exact floor depends on the number of classes, smoothing convention, logarithm base, class weights, and reduction. Calculate those before comparing a run with zero.

## Label smoothing minimum loss

Let q be the target distribution and p the model's predicted distribution. Cross-entropy is H(q, p) = negative sum over classes of qk log pk.

It can also be written as H(q) + KL(q || p). The entropy H(q) is fixed by the target, while KL divergence is never negative.

The minimum therefore occurs at p = q. The KL term reaches zero, but the cross-entropy remains H(q).

With a one-hot target, H(q) is zero. The model approaches that floor by sending the correct-class probability toward one and every other probability toward zero.

With label smoothing, q contains several positive values. Its entropy is positive, so the cross-entropy floor is positive too.

The gradient tells the same story. For softmax followed by cross-entropy, the logit gradient is p minus q. It vanishes at p = q even though the reported loss is not zero.

Fanout's[cross-entropy from logits guide](/blog/softmax-cross-entropy-from-logits)derives that gradient from the softmax normalization.

## PyTorch and Inception include the true class

The original[Inception label-smoothing definition](https://arxiv.org/abs/1512.00567)mixes the one-hot target with a uniform distribution over all C classes.

With smoothing epsilon, the correct-class target is t = 1 - epsilon + epsilon/C. Each other class receives r = epsilon/C.

[PyTorch CrossEntropyLoss](https://docs.pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html)documents the same mixture of ground truth and the uniform distribution.

Its minimum loss in natural-log units is:

Lmin = -t log t - (C - 1) r log r.

Nothing about this expression depends on the input example. For fixed C and epsilon, it is the per-example theoretical floor before class weighting or reduction changes the aggregate.

Calling t = 1 - epsilon is a common implementation mistake under this convention. The true class also receives its epsilon/C share from the uniform distribution.

For C = 10 and epsilon = 0.1, t is 0.91 and each other target is 0.01, not 0.9 and 0.0111.

## Calculate three floors

Use natural logarithms, which report the result in nats and match the common cross-entropy implementation.

For 2 classes and epsilon = 0.1, the target is 0.95 for the correct class and 0.05 for the other class.

The floor is negative 0.95 log 0.95 minus 0.05 log 0.05, which is about 0.1985.

For 10 classes, the target is 0.91 for the correct class and 0.01 for each of nine wrong classes.

The floor is negative 0.91 log 0.91 minus 9 times 0.01 log 0.01, which is about 0.5003.

For 1,000 classes, the correct target is 0.9001 and each of 999 wrong classes receives 0.0001.

The floor is about 1.0148. The loss is larger even though the correct-class target remains near 0.9 because the smoothing mass is spread across many terms.

This is why a language model or large classifier can show a smoothed training loss above one while sitting close to its target distribution.

The[good cross-entropy loss guide](/blog/what-is-a-good-cross-entropy-loss-value)uses log C as the uniform-prediction baseline. Label smoothing adds a different reference point: the target-entropy floor.

## The wrong-classes-only convention changes the answer

Some libraries and tutorials keep the correct target at 1 - epsilon and divide epsilon only among the C - 1 incorrect classes.

Under that convention, t = 1 - epsilon and r = epsilon/(C - 1).

The floor becomes negative (1 - epsilon) log(1 - epsilon) minus epsilon log(epsilon/(C - 1)).

For C = 10 and epsilon = 0.1, the correct target is 0.9 and each wrong target is about 0.01111. The minimum is about 0.5448, not 0.5003.

For binary classification, the difference is larger. The all-class convention gives targets 0.95 and 0.05 with a 0.1985 floor.

The wrong-class-only convention gives 0.9 and 0.1 with a 0.3251 floor.

Neither convention is a mathematical error if implemented consistently. The error is comparing losses or epsilon values without checking which target distribution the code constructs.

A small technical page on Google's first result page notices the nonzero floor, but most ranking explainers stop at intuition, benefits, or sample code. Few calculate both conventions across class counts.

## Positive loss can mean zero gradient

Suppose the 10-class model predicts exactly 0.91 for the correct class and 0.01 for every other class.

Its loss is 0.5003, and its logit gradient p - q is zero for all ten classes. The optimizer receives no request from this example to move closer to a one-hot output.

If the correct prediction rises to 0.99, the model is not closer to the smoothed target. It must take probability mass away from the wrong classes, so their negative log terms grow.

The loss rises after the prediction passes the softened optimum. Label smoothing is therefore more than a cap placed on a displayed confidence score.

It changes the target of optimization. The model can still output probabilities above t on particular examples, but that is not the per-example distribution that minimizes the smoothed objective.

A recent[selective-classification study](https://arxiv.org/abs/2403.14715)uses the same property in a binary example: smoothing 0.6 moves the optimum to scores 0.7 and 0.3.

## Convert the floor to perplexity carefully

Perplexity is exp of average cross-entropy when the loss uses natural logarithms.

Exponentiating the three all-class floors gives about 1.220 for 2 classes, 1.649 for 10 classes, and 2.759 for 1,000 classes.

Those are floors for perplexity computed from the smoothed training objective. They are not floors for hard-label evaluation perplexity.

Evaluation code often turns label smoothing off and scores the observed class directly. That metric can still approach zero cross-entropy and perplexity one on perfectly predicted data.

Do not compare a smoothed training perplexity with an unsmoothed validation perplexity as if they share one baseline. Record the target construction used by each metric.

The logarithm base also changes the number. Natural logs produce nats. Base-2 logs produce bits, and exponentiating with base two returns the matching perplexity.

## Reduction, weights, and ignored classes move the report

The closed-form floor above is per example with equal class weights and a valid target distribution over C classes.

PyTorch's default reduction averages losses. A sum reduction multiplies the batch result by the number of included examples, so its minimum grows with batch size.

Class weights alter the objective term by term. The minimizing prediction and floor can change because the weighted target no longer behaves like the simple unweighted q.

An ignore index removes selected examples from the reduced loss. Padding masks in language modeling do the same job at token level.

Mixup, distillation, or data-dependent soft labels create a different q for every example.

The minimum remains H(q) for ordinary unweighted cross-entropy. There is no single dataset floor until those target entropies are averaged.

The[NeurIPS label-smoothing study](https://proceedings.neurips.cc/paper/2019/hash/f1748d6b0fd9d439f71450117eba2725-Abstract.html)found that smoothing can improve calibration while making a trained teacher worse for knowledge distillation.

Loss floors do not tell you whether smoothing helps the downstream task. They only prevent a positive training loss from being misdiagnosed.

## A practical check for a training run

Read the implementation first. Construct the target vector for one example and print its values.

Confirm whether epsilon is spread across all C classes or only C - 1 wrong classes. Check that the probabilities sum to one.

Calculate H(q) with the same logarithm base, weights, mask, and reduction used by the loss. That is the comparison floor.

Then measure the gap between observed loss and the floor. The raw loss alone mixes irreducible target entropy with model mismatch.

For ordinary unweighted cross-entropy, observed loss minus H(q) is the average KL divergence from the smoothed target to the prediction.

That gap reaches zero only when the model matches q on every scored example. It measures the optimization gap without mixing in target entropy.

Use hard-label accuracy and unsmoothed evaluation loss alongside it. A model can match the smoothed objective and still generalize poorly, or sit above the theoretical floor while improving the metric that matters.

Label smoothing minimum loss is target entropy. C, epsilon, and the smoothing convention determine the number.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md