Skip to content

Underflow

See how positive probabilities become stored zeros, how subnormal numbers extend floating-point range, and why log probabilities must be computed before information is lost.

Underflow is a loss of numerical accuracy when a nonzero result is so close to zero that the chosen floating-point format cannot retain its usual relative precision. The stored result may be a small subnormal number or may round all the way to zero.

For a quant, this can turn a positive probability or likelihood into a stored zero. Comparing two models then becomes unreliable if both products disappear before their relative sizes are examined. A numerical zero does not establish that an event is impossible.

A positive product that disappears

Consider a synthetic model with 400 independent stages. At each stage, a specified event has probability p=0.1p=0.1. The probability that every specified event occurs is

P=p400=10−400>0.P=p^{400}=10^{-400}>0.
This is an arithmetic illustration, not a calibrated market forecast. The independence assumption is what permits multiplication; numerical technique cannot justify that modelling assumption.
In the checked Python binary64 calculation, both 0.1 ** 400 and sequential multiplication of 400 copies of 0.1 give 0.0. Even the nearest binary64 representation of the exact decimal value 10−40010^{-400} is zero. Binary64's initial approximation to 0.10.1 is a separate source of rounding error; it does not explain away the range failure.
For independent events of probability 0.1, sequential binary64 products are normal at 300 events, subnormal at 310 and 323, and zero at 324 and 400. The exact 400-event probability is positive, 10 to the minus 400; its natural logarithm is about minus 921.0340372.
For independent events of probability 0.1, sequential binary64 products are normal at 300 events, subnormal at 310 and 323, and zero at 324 and 400. The exact 400-event probability is positive, 10 to the minus 400; its natural logarithm is about minus 921.0340372.
Original dimensionless calculation, with positive stored values displayed to six significant figures. The mathematical probabilities remain positive in every row; the stored products lose accuracy and eventually become zero.

The last nonzero values are subnormal

Floating-point arithmetic has both a precision limit and a range limit. In IEEE binary64 with gradual underflow, the smallest positive normal number is
N=2−1022≈2.2250739×10−308.N=2^{-1022}\approx2.2250739\times10^{-308}.

That is not the smallest positive representable number. Subnormal numbers extend towards zero in fixed steps of

q=2−1074≈4.9406565×10−324.q=2^{-1074}\approx4.9406565\times10^{-324}.
The positive subnormals are q,2q,…,(252−1)qq,2q,\ldots,(2^{52}-1)q. As the magnitude falls, a fixed step becomes a larger fraction of the answer: relative accuracy deteriorates. This gradual loss is preferable to discarding every result below NN immediately. NumPy's machine-limits documentation distinguishes the two boundaries and warns about reduced subnormal precision.

A subnormal result can still be exact: N/2=2−1023N/2=2^{-1023} is representable. Merely producing a subnormal value does not necessarily raise an underflow signal. With round-to-nearest, ties-to-even, a positive exact result below q/2q/2 rounds to zero; the halfway value q/2q/2 also rounds to zero. Different rounding modes or settings that flush subnormals to zero change this behaviour.

Do not apply the familiar normal-range relative-error bound indiscriminately here. Rounding a positive result to zero gives 100% relative error, however tiny its absolute error is.

Take logs before multiplying

For positive factors, use the identity

log⁡ ⁣(∏i=1npi)=∑i=1nlog⁡pi.\log\!\left(\prod_{i=1}^{n}p_i\right)=\sum_{i=1}^{n}\log p_i.

In the example, the exact decimal inputs give

log⁡P=400log⁡(0.1)≈−921.0340371976183.\log P=400\log(0.1)\approx-921.0340371976183.
Python's 400 * math.log(0.1) produces the finite binary64 value -921.0340371976182. Keeping a log-likelihood lets a model-fitting routine compare products without constructing them. Logs preserve ordering for positive values; differences of logs represent log ratios.
Taking math.log after the product became 0.0 fails with ValueError in Python. The lost probability cannot be recovered from that zero. Converting the finite log result back with math.exp also gives 0.0 in this example: the log representation solves the storage problem for the log value, not for the original probability.

A true zero factor needs its own branch. For a discrete probability model, a factor assigned exactly zero gives a zero product and an extended-real log value of −∞-\infty when at least one factor is present. That differs from a positive product lost during computation. The empty product is 1 and its log is 0. These cases should be handled before evaluating logarithms.

Keep the numerical and modelling decisions separate

If a task only needs log probabilities, rankings or likelihood ratios, retain the log representation. If it needs the unscaled probability, use a representation with adequate range and stated precision, or report the range failure. Repeated rounded multiplication can also stagnate at a positive subnormal value, so a nonzero output alone is no guarantee of accuracy.

Warning controls help reveal the failure. NumPy allows underflow to warn or raise an error; changing the warning policy does not restore discarded information. Replacing every tiny value with an arbitrary probability floor changes the calculation. Treating a contribution as negligible instead requires an error budget appropriate to the task.
The membership lesson Numerical Conditioning and Stability places normal-range arithmetic error alongside hedge, regression and covariance sensitivity. Its public page provides a preview. For large magnitudes, the corresponding range failure is overflow.

Sources

Last updated
  • numerical methods
  • floating point
  • underflow
  • log probability
Keep connecting the ideas
← Back to the glossaryBack to top ↑