Skip to content

Numerical Precision

How finite numerical detail affects cash flows and risk calculations: binary floats, decimal inputs, naive summation, conditioning and tolerance choices.

Numerical precision is the amount of detail a number representation or arithmetic system can retain. In quant work, it limits how finely prices, cash flows and risk sensitivities can be represented and combined. A calculation can print many digits while losing the economically relevant difference between two large values.
Precision is distinct from accuracy, which measures closeness to the intended reference answer. More precision gives a computation more room to work; it does not correct a wrong model, bad market data or an unstable formula.

Four different questions

QuestionConceptWhat to investigate
How much numerical detail can be retained?PrecisionRepresentation and arithmetic settings
How close is the result to the reference?AccuracyError against an independent benchmark
How sensitive is the exact answer to small input changes?ConditioningThe problem and its input uncertainty
How well does the computation control rounding effects?Numerical stabilityThe algorithm and its intermediate calculations

Rounding is the act of replacing a value with one from a permitted set. It can happen during storage and arithmetic, or deliberately when a contractual payment is rounded to a currency's settlement increment. Those are separate decisions.

Decimal-looking inputs, binary storage

On usual modern platforms, Python's float uses binary64 with 53 bits of significand precision. This does not promise 53 decimal digits, or a fixed number of correct digits in every computed answer. Many decimal fractions have no finite binary representation. Python's floating-point tutorial explains the conversion.
from decimal import Decimal print(0.1 + 0.2) print(Decimal("0.1") + Decimal("0.2")) print(Decimal(0.1) == Decimal("0.1")) # 0.30000000000000004 # 0.3 # False
The first result reflects binary approximations and rounding. The second adds decimal inputs exactly in this case. The third exposes a construction trap: Decimal(0.1) converts the already approximated float exactly; it cannot recover the decimal input you intended. Construct decimal amounts from the original strings, or from exact integers with an explicit scale. Decimal arithmetic still has a finite arithmetic context: division by three, for example, generally needs rounding.
Displaying the first result as 0.30 changes its appearance, not the stored value or earlier calculations. Decimal arithmetic is useful for specified monetary rules; binary floats remain appropriate for many pricing and statistical calculations.

A small amount can vanish before the final subtraction

Consider three fictional, same-currency cash flows, in this observation order: USD 101610^{16}, USD 11, and USD −1016-10^{16}. Their exact sum is USD 11.

import math cash_flows = [1e16, 1.0, -1e16] total = 0.0 for cash_flow in cash_flows: # Deliberately naive left-to-right accumulation. total += cash_flow print(total) print(math.fsum(cash_flows)) print(math.ulp(1e16)) # 0.0 # 1.0 # 2.0

Near 101610^{16}, adjacent binary64 values are two units apart. Adding one lands halfway between representable values and rounds back to 101610^{16} under the usual ties-to-even rule. The final subtraction produces zero because the one was already lost. This extreme scale example makes a mechanism visible; it is not a typical cash-flow size.

Exact prefix totals are 10000000000000000, 10000000000000001 and 1 USD; naive binary64 totals are 10000000000000000, 10000000000000000 and 0 USD. math.fsum returns 1 USD.
Exact prefix totals are 10000000000000000, 10000000000000001 and 1 USD; naive binary64 totals are 10000000000000000, 10000000000000000 and 0 USD. math.fsum returns 1 USD.
The USD 1 disappears in the second naive addition. The table compares cumulative totals, not values on a common-size plot. Cash flows are fictional and processed in the stated order.
math.fsum retains multiple partial sums to reduce summation error. It returns the correct result here; it does not make all floating-point work exact or repair inaccurate inputs. This loop is explicitly naive: Python's built-in sum gained improved float summation in Python 3.12 and returns 1.0 for this input on the Python 3.14.4 runtime checked here.

Choose an error budget in the right units

For approximate comparisons, decide how much error matters economically. An absolute tolerance has the same units as the result; a relative tolerance is dimensionless. Near zero, a relative tolerance alone is usually insufficient. Python's math.isclose accepts the larger of its absolute and relative allowances.
import math residual_usd = 0.004 print(math.isclose(residual_usd, 0.0, rel_tol=0.0, abs_tol=0.005)) print(math.isclose(residual_usd, 0.0)) # True # False

Here a fictional reconciliation permits a half-cent absolute residual. The chosen rule passes USD 0.004; a default relative comparison to zero fails. This tolerance is a validation policy, not a settlement-rounding rule. Exact equality remains appropriate when exact identity is the intended test.

Keep units explicit: changing dollars to millions also changes the numerical absolute tolerance. Scaling can improve intermediate numerical ranges, but cannot recreate discarded information or resolve underlying sensitivity. For offsetting totals, use a suitable summation routine; for exact fixed-increment accounting, consider integer minor units or decimal arithmetic with an explicit rounding rule. For ill-conditioned models, investigate the inputs and formulation as well as precision.

For a deeper distinction between problem sensitivity and algorithm behaviour, read Numerical Conditioning and Stability. Testing Numerical Code develops reference values, invariants and tolerance-based tests. Both offer public previews; full lessons require membership.

Sources

The examples were executed with Python 3.14.4. The linked official documentation was Python 3.14.10 when checked.

Last updated
  • numerical methods
  • python
  • floating point
  • decimal arithmetic
  • rounding
  • numerical stability
← Back to the glossaryBack to top ↑