Label smoothing vs confidence penalty

Label smoothing vs confidence penalty is a comparison between two ways to discourage overconfident predictions. Label smoothing changes the target distribution. Confidence penalty adds the model's negative entropy to the loss.

Both pull predictions toward a uniform distribution, but in opposite KL directions. Label smoothing corresponds to KL(uniform || prediction). Confidence penalty corresponds to KL(prediction || uniform).

Reversing the arguments changes the loss near zero probability and changes the gradient on each logit. The methods are related, not interchangeable.

Label smoothing vs confidence penalty

The original Inception label-smoothing section replaces a one-hot target with a mixture of that target and a prior distribution, usually uniform.

For K classes and smoothing strength epsilon, the target is q = (1 - epsilon) times one-hot(y) plus epsilon times uniform.

Training still minimizes cross-entropy, but against q instead of the hard label. Every class now carries some target probability.

The confidence penalty paper leaves the hard target intact. It subtracts beta times the prediction entropy H(p) from negative log-likelihood.

Because high entropy lowers that objective, the model pays a penalty for placing nearly all probability on one class.

Fanout's softmax cross-entropy explainer derives the ordinary p minus y gradient. Label smoothing changes y to q; confidence penalty adds a second gradient that depends on p itself.

Label smoothing changes the supervised target

Suppose a three-class example belongs to class 0 and epsilon is 0.1. Under the common uniform-mixture definition, q is [0.9333, 0.0333, 0.0333].

The correct class keeps most of the target mass. Each incorrect class receives 0.0333, regardless of whether it is a plausible alternative for this example.

The cross-entropy gradient with respect to logit j is p_j minus q_j. That fixed q_j matters when an incorrect class has probability near zero.

If p_j approaches zero, its gradient approaches minus epsilon divided by K. Gradient descent therefore raises that logit rather than letting its probability vanish.

This behavior comes from the target, not from a separate entropy calculation. Frameworks can implement label smoothing inside cross-entropy without reading the other predicted probabilities beyond the usual softmax.

It also changes the supervised term's optimum. With enough capacity and repeated identical labels, cross-entropy now asks for q rather than a one-hot distribution.

The good cross-entropy loss guide explains why the resulting training-loss floor depends on the target entropy.

Confidence penalty changes the objective

Confidence penalty uses L = -log p_y - beta H(p). The target remains one-hot in the first term, while the second term rewards higher entropy across the whole prediction.

The entropy gradient for logit j depends on p_j times a centered log probability. For the penalty term, it is beta times p_j times (log p_j + H(p)).

Unlike label smoothing, that gradient is adaptive. A class with moderate probability can receive a larger entropy gradient than a class whose probability is already extremely small.

As p_j approaches zero, p_j log p_j also approaches zero. Confidence penalty does not maintain a fixed upward gradient on every vanishing class.

The original paper tested the methods across image classification, language modeling, translation, and speech recognition. Which one won depended on the task and the separately tuned coefficient.

That empirical result fits the math. Epsilon and beta do not express the same amount of regularization, so equal numeric values are not a controlled comparison.

The KL direction is the core difference

The 2020 generalized entropy regularization paper places both methods in one family.

Up to constants and coefficient reparameterization, label smoothing adds KL(u || p), where u is uniform. Confidence penalty adds KL(p || u).

KL(u || p) averages log(u_j / p_j) under the uniform distribution. Every class receives weight 1/K before the model probability appears inside the logarithm.

KL(p || u) averages log(p_j / u_j) under the model distribution. A class receives little weight when the model already assigns it little probability.

The first direction strongly resists missing support that exists in u. The second is more tolerant of probabilities approaching zero.

Neither direction knows which wrong classes are semantically similar. Uniform label smoothing treats every incorrect class alike, while confidence penalty responds only to the current probability shape.

A non-uniform prior can change the label-smoothing side. It does not remove the directional difference between prior-to-model and model-to-prior KL.

A worked three-class example

Take p = [0.98, 0.01, 0.01] and a uniform distribution u = [1/3, 1/3, 1/3]. The prediction is correct for class 0 but highly concentrated.

KL(u || p) is about 1.978 nats. KL(p || u) is about 0.987 nats. They differ even before either value is multiplied by a coefficient.

Now use epsilon = 0.1. The smoothed target q is [0.9333, 0.0333, 0.0333].

The label-smoothed cross-entropy gradient p - q is approximately [0.0467, -0.0233, -0.0233]. Both small classes receive the same fixed upward push.

The unweighted confidence-penalty gradient is approximately [0.0899, -0.0449, -0.0449]. Multiplying by beta scales this vector before it is added to the hard-label gradient.

The two gradient vectors happen to be proportional in this symmetric example. Break the symmetry, and confidence penalty weights classes through their current p_j while label smoothing keeps the fixed q_j targets.

For p = [0.98, 0.019, 0.001], label smoothing still targets 0.0333 for both wrong classes. Confidence penalty gives the 0.001 class a much smaller entropy gradient than the 0.019 class.

Matching one example's gradient by tuning beta does not make the objectives equivalent across a dataset or across training steps.

Zero probabilities expose the difference

Let one predicted class probability shrink toward zero while uniform u keeps mass 1/K on it.

In KL(u || p), the term (1/K) log((1/K) / p_j) grows without bound. Label smoothing therefore assigns infinite cost to an exact zero under the mathematical softmax limit.

In KL(p || u), the term p_j log(p_j / (1/K)) approaches zero. The entire divergence remains bounded and approaches log K for a one-hot prediction.

The generalized entropy study identifies this support behavior as a reason label smoothing can be undesirable for language generation, where nearly impossible tokens may reasonably receive vanishing probability.

That does not make confidence penalty universally better. It states a concrete modeling choice: label smoothing insists on support from its prior, while confidence penalty mainly discourages low total entropy.

The NeurIPS label-smoothing study found a separate tradeoff.

Smoothing improved calibration but erased logit information useful for knowledge distillation.

Epsilon and beta need separate tuning

Label smoothing with epsilon changes two pieces at once. It scales the hard-target cross-entropy by 1 - epsilon and adds epsilon times cross-entropy from the prior.

To express it as ordinary hard-target loss plus a KL regularizer, factor out 1 - epsilon. The relative regularizer weight becomes epsilon divided by 1 - epsilon, apart from constants.

Confidence penalty adds beta times KL(p || u), again apart from a constant. It does not rescale the hard-target term unless the implementation does so separately.

Setting epsilon = beta = 0.1 therefore does not produce equal gradients or equal loss contributions.

Compare each method against the same hard-label baseline. Keep optimizer, schedule, augmentation, and model fixed, then tune its own coefficient on the metric the application uses.

Log accuracy, hard-label negative log-likelihood, calibration, mean entropy, and maximum predicted probability. Do not judge either regularizer only by the training loss it changes.

Choosing between the two objectives

Choose label smoothing when a fixed prior is part of the intended target and you want a nonzero gradient against every class in that prior's support.

Inspect the exact framework definition. Some libraries spread epsilon across all K classes, while others spread it only across the K - 1 incorrect classes.

Choose confidence penalty when you want to regularize the prediction's entropy without assigning a fixed target mass to every wrong class.

It requires an explicit entropy term in many training stacks. Verify its sign: subtract entropy from the minimized loss, or equivalently add KL(p || u) after dropping the constant.

For a teacher model, test distillation quality rather than assuming improved calibration will help the student. For language generation, inspect whether forced support on unlikely tokens matches the task.

The ML math curriculum develops cross-entropy, entropy, and KL divergence from the same probability identities. The practical distinction is the direction: u to p fixes support, while p to u weights the model's current beliefs.