Skip to content

Catastrophic cancellation

Why subtracting nearly equal approximate quantities can erase a small answer, illustrated by a cosine deficit and portfolio-mark variance, with practical numerical repairs.

Catastrophic cancellation is a severe loss of relative accuracy when nearly equal approximate quantities are subtracted: their common leading digits disappear, exposing errors already present in the operands. A small absolute error that was harmless on the operands' scale can overwhelm their much smaller difference.

For a quant developer, the warning sign is a small quantity extracted from large intermediate values: a variance from raw moments, a price increment from two nearby prices, or a hedge residual from large exposures. The formula may be correct in real arithmetic while its evaluation loses the quantity of interest.

The subtraction may be blameless

Suppose the desired quantities are aa and bb, but earlier calculations produce a^=a+ea\widehat a=a+e_a and b^=b+eb\widehat b=b+e_b. Even if their subtraction is exact,

a^−b^=(a−b)+(ea−eb).\widehat a-\widehat b=(a-b)+(e_a-e_b).

For a nonzero true difference, the resulting relative error is

∣ea−eb∣∣a−b∣.\frac{\lvert e_a-e_b\rvert}{\lvert a-b\rvert}.

The denominator is the danger. Small errors in the operands need not be small compared with the difference. Their errors might also cancel; a large error is a possibility, not an inevitable consequence of every subtraction.

Indeed, subtracting nearby exactly represented numbers can be exact. Sterbenz's lemma states that for positive floating-point numbers aa and bb with a/2≤b≤2aa/2\le b\le2a, their difference is exactly representable in the same format, assuming the arithmetic supports the required small results, including gradual underflow where needed. Under correctly rounded subtraction, no new rounding error is then introduced. The familiar instruction “never subtract nearby numbers” is too crude: examine how those numbers were obtained. Goldberg's floating-point tutorial distinguishes this benign subtraction from the exposure of earlier rounding errors.

A small answer that becomes zero

Consider f(x)=1−cos⁡xf(x)=1-\cos x at x=10−8x=10^{-8} radians. The exact value is approximately 5×10−175\times10^{-17}. In the checked Python binary64 run below, cos(x) rounds to 1.0, so subtracting it from 1 returns zero. That final subtraction is exact for its stored operands; the cosine's small deficit has already disappeared.

The identity

1−cos⁡x=2sin⁡2(x/2)1-\cos x=2\sin^2(x/2)

provides a better evaluation near zero. It forms the small quantity directly rather than recovering it from a rounded value close to 1.

import math x = 1e-8 direct = 1 - math.cos(x) reformulated = 2 * math.sin(x / 2)**2 # direct: 0.0 # reformulated: 5.0000000000000005e-17

The direct result has 100% relative error in this run. Against a 120-digit reference for the actual stored input, the reformulated result has relative error below 10−1610^{-16}. The decimal input 10−810^{-8} is itself rounded slightly when stored in binary64; comparing both evaluations against the same stored-input reference isolates their computational error. Specific outputs depend on the mathematical library, precision and evaluation order; a binary64 label alone does not specify every transcendental-function result.

In checked binary64 calculations, 1 minus cos of 1e-8 gives zero while 2 sin squared of half the argument gives about 5e-17. For three portfolio marks from USD 100,000,001 to USD 100,000,003, raw-moment population variance gives zero while centring gives approximately 2/3 USD squared.
In checked binary64 calculations, 1 minus cos of 1e-8 gives zero while 2 sin squared of half the argument gives about 5e-17. For three portfolio marks from USD 100,000,001 to USD 100,000,003, raw-moment population variance gives zero while centring gives approximately 2/3 USD squared.
Original synthetic examples. Both direct evaluations return zero in the checked runtime; the corresponding reference quantities are approximately 5×10−175\times10^{-17} and exactly 2/32/3 USD². The diagram separates the rounded intermediate quantities from the final subtraction.

A risk calculation with the same structure

Take three synthetic portfolio marks: USD 100,000,001, USD 100,000,002 and USD 100,000,003. Their mean is USD 100,000,002. Treat the three marks as the complete collection whose dispersion we want; the population variance uses divisor 3, not the divisor 2 of an unbiased sample-variance estimator.
Two exact-arithmetic expressions for this variance are
v=13∑i=13xi2−xˉ2=13∑i=13(xi−xˉ)2.v=\frac{1}{3}\sum_{i=1}^3 x_i^2-\bar x^2 =\frac{1}{3}\sum_{i=1}^3(x_i-\bar x)^2.

The deviations are USD −1-1, USD 0 and USD 1, so v=2/3v=2/3 USD². Adding a common level to every mark changes the mean but leaves this exact dispersion unchanged.

With ordinary Python float operations, however, sum(x*x for x in marks)/3 and mean*mean both evaluate to 1.0000000400000004e16. Subtracting them gives zero. Computing the mean first and then summing the squared deviations gives 0.6666666666666666 USD². The integer marks are all exactly representable in binary64; the raw-moment route loses information in its intermediate arithmetic.

This is a numerical demonstration, not an estimate of market volatility from three observations. For real risk data, choose a tested centred, two-pass or online variance algorithm, and consider compensated summation where accumulation error matters. These methods still have finite precision and range limits. They cannot recover mark differences lost during ingestion or fix an inappropriate statistical model. A raw-moment result can even become negative through rounding; silently replacing it with zero hides the diagnostic.

Repair the calculation, and identify what cannot be repaired

Numerical stability concerns errors introduced by the computation. Conditioning concerns sensitivity of the exact answer to specified input perturbations. These must be distinguished before choosing a remedy.

As a function of its single input xx, the cosine deficit has relative condition number

κf(x)=∣xsin⁡x1−cos⁡x∣⟶2(x→0, x≠0).\kappa_f(x)=\left|\frac{x\sin x}{1-\cos x}\right|\longrightarrow2 \qquad(x\to0,\ x\ne0).

Thus the small nonzero result does not itself make this mathematical problem relatively ill-conditioned near zero. The direct evaluation introduces a fragile intermediate subtraction that the identity avoids. By contrast, if two independently uncertain portfolio values are the original inputs and their difference is the required answer, their uncertainty may already exceed that difference. Algebra cannot manufacture missing input information.

A useful repair follows the source of the error: reformulate a fragile identity, use specialised functions such as expm1 or log1p for small exponential or logarithmic increments, centre a variance calculation, or increase working precision before the damaging intermediate steps. Verify representative cases against a sufficiently accurate independent reference. At an exact zero, including x=0x=0 in the cosine example, relative error is undefined; assess absolute error instead.
For the wider connection to hedge ratios and covariance calculations, Numerical Conditioning and Stability develops the distinction through matrix problems. Its full lesson requires membership; the linked page provides a public preview.

Sources

Last updated
  • numerical methods
  • floating point
  • rounding
  • variance
  • stability
Keep connecting the ideas
← Back to the glossaryBack to top ↑