Convert class scores to probabilities, compute log-sum-exp without overflowing, and distinguish harmless score shifts from temperature scaling and calibration.
Softmax and log-sum-exp
Softmax converts a finite vector of real scores into positive weights that sum to one. In a classifier, these weights represent a model's probabilities for mutually exclusive classes. Log-sum-exp is the natural logarithm of the sum of the scores' exponentials: the scalar normalising quantity behind softmax. Computing both carefully lets a quant model produce class probabilities and likelihood losses even when the raw exponentials exceed the computer's range.
For K≥2 finite real scores, or logits, z1,…,zK, define
LSE returns one number; softmax returns a vector. Their connection is also differential: ∂L/∂zi=pi. The largest score receives the largest probability, but softmax does not select a single winner.
The level of the scores is irrelevant
Imagine a fictional classifier assigning each trading day to exactly one of three classes, A, B or C. Its dimensionless logits are 1000, 1001 and 1002. These are scores, not returns or probabilities.
Directly evaluating e1002 causes overflow in standard binary64 arithmetic. Yet the desired probabilities are perfectly ordinary. Subtract the largest score, m=1002, before exponentiating:
Class
Original score
Score minus maximum
Shifted exponential
Probability
A
1000
−2
0.135335
0.090031
B
1001
−1
0.367879
0.244728
C
1002
0
1.000000
0.665241
The shifted exponentials sum to approximately 1.503215. Dividing each by this sum gives probabilities that sum to one, apart from rounding. Subtracting any common constant leaves softmax unchanged because the common exponential factor cancels. Choosing the maximum makes every exponential argument nonpositive and at least one zero:
pi=∑jezj−mezi−m,L=m+log(j∑ezj−m).
For these scores, L≈1002.4076059644. Unlike softmax, LSE changes when a common constant is added: it rises by that constant. More generally, m≤L≤m+logK, so LSE is a smooth approximation to the maximum. It is also useful for adding positive quantities already stored as logarithms, without first reconstructing their potentially enormous or tiny values.
Temperature changes the gaps
A common shift preserves every score difference. Dividing scores by a positive temperatureT changes those differences and generally changes the probabilities:
At T=2, the example becomes approximately 0.186324, 0.307196 and 0.506480. The ranking stays the same, but the distribution is flatter. Temperature is therefore a modelling choice, whereas subtracting the maximum is an algebraically equivalent way to perform the same calculation.
For fictional logits 1000, 1001 and 1002, subtracting the maximum gives −2, −1 and 0, and probabilities 9.00%, 24.47% and 66.52%. Dividing logits by temperature 2 instead gives 18.63%, 30.72% and 50.65%. Log-sum-exp is approximately 1002.4076.
Original fictional example: shifting preserves the probability distribution; temperature scaling changes it. The plotted bars use unrounded calculations.
INTERACTIVE EXPLORER
Stable softmax and temperature
Shift large scores safely and inspect probabilities and gradients.
Stay in log space when the loss needs logarithms
The maximum shift improves numerical stability; it does not eliminate every floating-point limitation. Very negative shifted scores can still produce underflow, and finite precision can erase small contributions.
For logits 0 and −1000, binary64 softmax rounds the second probability to zero even though its mathematical value is positive. Taking the logarithm afterwards gives negative infinity. A dedicated log-softmax calculation instead returns approximately 0 and −1000. Compute the log probabilities through the shifted expression
logpi=(zi−m)−log(j∑ezj−m),
rather than taking the logarithm of a probability already rounded to zero. For an observed class c, its negative log-likelihood is −logpc=L−zc; the shifted form avoids constructing the probability first. A library's log-softmax or loss accepting logits implements this idea with numerical refinements. When processing a batch, normalise over the class axis for each observation, not across observations. SciPy's default axis is None, which normalises the whole input array.
The softmax and cross-entropy derivation develops the corresponding training gradient. A public preview is available; the full lesson requires membership.
A probability constraint is not a validation result
Normalisation alone does not establish probability calibration: predictions near 70% need not occur about 70% of the time. Temperature adjustments require validation too.
Softmax also imposes competition among its outputs. For multilabel classification, where several labels may hold together, separate sigmoid outputs can estimate labelwise marginal probabilities without making them sum to one. This does not establish independence between the labels.
Likewise, positive weights summing to one can parameterise a fully invested, long-only portfolio. Softmax alone supplies neither an investment objective nor expected returns, risks, costs or optimal allocations. The interpretation comes from the model and its evidence, not the normalisation formula.
Sources
SciPy: softmax — definition, class-axis choice and shifted implementation.
