Skip to content

Label smoothing

How label smoothing mixes a hard class target with a reference distribution, changes cross-entropy training, and differs from calibration and distillation.

Label smoothing is a training regularisation method that replaces a hard class target with a mixture of that target and a reference distribution. For a quant classifier assigning each observation to one mutually exclusive return category, it changes how strongly training rewards certainty about the observed category. It does not change the observation's recorded label.

Specify where the smoothing mass goes

Consider K≥2K\ge2 mutually exclusive classes. Let yky_k be the one-hot target: one for the observed class, zero for the others. Under the uniform-mixture convention used by Szegedy et al. (2016), §7, the training target is
y~k=(1−ε)yk+εK,0≤ε≤1.\widetilde y_k=(1-\varepsilon)y_k+\frac{\varepsilon}{K}, \qquad 0\le\varepsilon\le1.

The uniform component includes the observed class. Its target weight is therefore 1−ε+ε/K1-\varepsilon+\varepsilon/K; every other class receives ε/K\varepsilon/K. The weights sum to one. Zero smoothing recovers the hard target; at ε=1\varepsilon=1 the target is entirely uniform and discards the class information.

This convention matters. Another convention assigns 1−ε1-\varepsilon to the observed class and spreads ε\varepsilon over only the other K−1K-1 classes. The same numerical smoothing parameter then means a different target. Always state the target construction when comparing experiments.

One observation, two loss values

Take a fictional three-class observation with class A recorded and ε=0.10\varepsilon=0.10. The uniform-mixture target becomes

y=(1,0,0),y~=(1415,130,130)≈(0.933333,0.033333,0.033333).y=(1,0,0),\qquad \widetilde y=\left(\frac{14}{15},\frac{1}{30},\frac{1}{30}\right) \approx(0.933333,0.033333,0.033333).
It is not (0.90,0.05,0.05)(0.90,0.05,0.05), which follows the other convention above. These are chosen training weights, not estimates that this observation has a 93.33% chance of truly belonging to A.

Keep the model prediction fixed at p=(0.80,0.10,0.10)p=(0.80,0.10,0.10). For a probability vector with positive entries and ∑kpk=1\sum_k p_k=1, the smoothed cross-entropy per observation is

Lsmooth=−∑k=1Ky~kln⁡pk.\mathcal L_{\mathrm{smooth}}=-\sum_{k=1}^K\widetilde y_k\ln p_k.
Using natural logarithms, the loss is measured in nats:
Lhard=−ln⁡(0.80)≈0.2231436,Lsmooth=−1415ln⁡(0.80)−130ln⁡(0.10)−130ln⁡(0.10)≈0.3617730.\begin{aligned} \mathcal L_{\mathrm{hard}}&=-\ln(0.80)\approx0.2231436,\\ \mathcal L_{\mathrm{smooth}}&=-\frac{14}{15}\ln(0.80) -\frac{1}{30}\ln(0.10)-\frac{1}{30}\ln(0.10) \approx0.3617730. \end{aligned}

The larger value does not demonstrate that the model became worse: its prediction has not moved. We changed the scoring objective. Training losses computed against different smoothing targets should not be treated as the same evaluation metric.

The hard target for classes A, B and C is 1, 0, 0; the smoothed target is 0.9333, 0.0333, 0.0333; the fixed prediction is 0.8, 0.1, 0.1. Natural-log loss changes from 0.2231436 to 0.3617730 nats per observation.
The hard target for classes A, B and C is 1, 0, 0; the smoothed target is 0.9333, 0.0333, 0.0333; the fixed prediction is 0.8, 0.1, 0.1. Natural-log loss changes from 0.2231436 to 0.3617730 nats per observation.
In this fictional example, smoothing changes the training target while the displayed model probabilities stay fixed.

What the objective encourages

With positive smoothing, assigning an arbitrarily tiny probability to any class incurs a penalty because every target weight is positive. For one freely fitted observation, cross-entropy is minimised at p=y~p=\widetilde y, rather than at the hard target. This is an objective-level statement; a model sharing parameters across many observations need not fit each target exactly. Label smoothing is not a hard cap on the model's predictions.

For softmax logits zkz_k, the loss gradient is ∂L/∂zk=pk−y~k\partial\mathcal L/\partial z_k=p_k-\widetilde y_k. Smoothing changes the target in this error signal. The SelfMadeQuant lesson Backward Functions for Basic Modules and Vectorized Backpropagation derives the hard-target softmax/cross-entropy calculation that this extends. Its preview is public; the full lesson requires membership.

Softer targets do not guarantee reliable probabilities

Müller, Kornblith and Hinton (2019), §3 report improved calibration in their image-classification and translation experiments. This is empirical evidence in those settings, not a guarantee for financial forecasts. Less extreme confidence can still be miscalibrated. If probabilities guide a decision, assess probability calibration against held-out observed outcomes, not against the artificially smoothed targets. Choose the smoothing strength using validation data and preserve a final test; financial observations also require an evaluation split that respects time and label availability.

Two related techniques act differently:

  • Temperature scaling rescales logits before softmax; when used for post-training calibration, its parameter is fitted on held-out data. Label smoothing changes the training targets.
  • Knowledge distillation uses a teacher model's outputs as targets, retaining example-specific information. Uniform label smoothing supplies the same reference distribution to every observation. Müller et al., §4, also found that smoothing a teacher could impair distillation, even when the teacher's accuracy improved.

Neither technique turns a heuristic target distribution into evidence about uncertainty in the recorded labels. Suspected label errors require investigation of how the labels were constructed.

Sources

Last updated
  • machine learning
  • regularisation
  • classification
  • soft targets
Keep connecting the ideas
← Back to the glossaryBack to top ↑