Knowledge distillation: Definition & Examples

Knowledge distillation trains a student model to imitate a teacher model's behaviour. The teacher may be a large neural network or an ensemble; the student is often smaller and cheaper to run. For a quant team, the attraction is practical: a costly research model might supervise a compact model for repeated forecasts. Whether useful predictive behaviour survives the transfer must be measured.

Teach the distribution, not just the winning class

In the classic output-based method, the teacher supplies soft targets: a probability for every class, rather than only a hard class label. Two observations can share the same winning class but have different runner-up probabilities. Keeping those differences gives the student a richer imitation task. The teacher is held fixed while the student learns; matching its architecture or copying its weights is unnecessary. This is the setup developed by Hinton, Vinyals and Dean (2015), §§1–2.

Consider one fictional two-class observation. At temperature 1, the teacher gives $p=(0.25,0.75)$ while the student gives $q=(0.50,0.50)$ . A hard teacher label would retain only “class B”. The soft target retains the teacher's uncertainty about class A.

Using natural logarithms, the teacher-to-student relative entropy, or KL divergence, is

D_{\mathrm{KL}}(p\Vert q) =0.25\log\frac{0.25}{0.50}+0.75\log\frac{0.75}{0.50} \approx0.130812\text{ nats}.

The soft-target cross-entropy has a different value:

H(p,q)=-\sum_i p_i\log q_i=\log2\approx0.693147\text{ nats}.

Their relationship is $H(p,q)=H(p)+D_{\mathrm{KL}}(p\Vert q)$ , where the teacher's entropy is $H(p)\approx0.562335$ nats. Since that term is constant with respect to the student, minimising either loss gives the same student gradients when the teacher, temperature and reduction convention are fixed. Their logged loss values are not interchangeable. Even perfect imitation, $q=p$ , gives zero KL but cross-entropy $H(p)$ .

Temperature changes the teaching signal

For teacher logits

v_i

and student logits

z_i

, apply the same temperature

T>0

to both softmax distributions:

p_i^{(T)}=\frac{e^{v_i/T}}{\sum_j e^{v_j/T}}, \qquad q_i^{(T)}=\frac{e^{z_i/T}}{\sum_j e^{z_j/T}}.

In our example, teacher logits $(0,\log3)$ produce $(0.25,0.75)$ at $T=1$ . At $T=2$ , they produce

p^{(2)}=\left(\frac{1}{1+\sqrt3},\frac{\sqrt3}{1+\sqrt3}\right) \approx(0.366025,0.633975).

The student logits $(0,0)$ still give $(0.50,0.50)$ . Higher temperature softens the teacher's probabilities while preserving their ranking. These softened values are training targets, not a claim that the true event probabilities have changed.

Teacher probabilities change from 0.25 and 0.75 at temperature 1 to 0.3660 and 0.6340 at temperature 2; the student remains at 0.50 and 0.50. At temperature 1, cross-entropy 0.6931 equals teacher entropy 0.5623 plus KL divergence 0.1308, all in nats.

A fictional binary example separates temperature softening from the constant difference between cross-entropy and KL divergence.

When observed labels are available, one common convention combines the usual label loss at temperature 1 with a distillation term:

\mathcal{L}=a\,H(y,q^{(1)})+b\,T^2D_{\mathrm{KL}}(p^{(T)}\Vert q^{(T)}), \qquad a,b\ge0.

Here

y

is a one-hot observed label and

a,b

are chosen weights; they need not sum to one. The official PyTorch tutorial's output-loss example uses this type of weighted objective. Its soft term is KL divergence despite the broader use of cross-entropy terminology around distillation.

The $T^2$ factor is a scaling convention, not a universal definition. For a fixed teacher, the unscaled soft cross-entropy gradient with respect to student logit $z_i$ is $(q_i^{(T)}-p_i^{(T)})/T$ . When temperature is large relative to fixed logit differences, the probability difference itself shrinks roughly as $1/T$ , motivating the compensating factor. It does not make different temperatures equivalent. In the classic setup the trained student returns to temperature 1 for prediction; this is separate from probability calibration.

A teacher belongs inside the research cutoff

For a hypothetical return-direction classifier, fitting the teacher on the entire history and then using its soft labels to train a student for an earlier backtest can transmit future information through the targets. Restricting only the student's inputs does not repair that contamination. At each historical fit cutoff, teacher fitting, preprocessing and soft-target generation must respect the information available then; held-out validation or test outcomes must not enter training through either model.

Use validation to choose the temperature, loss weights and student design, then preserve a final test of the selected procedure. Compare the student with the teacher and with the same student trained directly on observed labels. Assess predictive quality, calibration when probabilities drive decisions, and actual inference cost separately. A smaller model is not automatically faster on every runtime, and imitation does not guarantee a better, truer or better-calibrated forecast.

The SelfMadeQuant lesson Time-Ordered Train, Validation and Test Design develops the cutoff and label-availability discipline behind this evaluation. The preview is public; the full lesson requires membership.

Sources

Hinton, Vinyals and Dean (2015), Distilling the Knowledge in a Neural Network, §§1–2.1: soft targets, shared temperature, weighted objectives and the high-temperature gradient approximation.
PyTorch, Knowledge Distillation Tutorial, “Knowledge distillation run”: fixed teacher, student-only updates and a weighted KL-plus-label-loss implementation.

Teach the distribution, not just the winning class

Temperature changes the teaching signal

A teacher belongs inside the research cutoff

Sources

Related terms

Neural network

Softmax and log-sum-exp