Skip to content

Maximum entropy principle

Choose a probability distribution by maximising entropy subject to explicit support, reference measure and constraints. A mean-two waiting-time example shows the result and its limits.

The maximum entropy principle chooses the probability distribution with the largest entropy among those satisfying specified information constraints, relative to a chosen reference measure. For a quant building a scenario or waiting-time model from limited information, it supplies an explicit baseline while making the modelling assumptions visible.
The inputs matter as much as the optimisation: which outcomes are allowed, how probability density is measured, and which facts must the distribution reproduce? “Use maximum entropy” is incomplete without them.

One mean, many possible waiting-time laws

Suppose a hypothetical market-event wait is nonnegative and has mean two time units. We seek a continuous density, with no point masses. Let xx denote the numerical wait in the chosen time unit, and use ordinary length measure dxdx (Lebesgue measure) on [0,∞)[0,\infty).

Among densities with finite, well-defined differential entropy, solve

max⁡f  h(f)=−∫0∞f(x)ln⁡f(x) dx,f(x)≥0,∫0∞f(x) dx=1,∫0∞xf(x) dx=2.\max_f\;h(f)=-\int_0^\infty f(x)\ln f(x)\,dx, \qquad f(x)\ge0,\quad \int_0^\infty f(x)\,dx=1,\quad \int_0^\infty xf(x)\,dx=2.
The answer is the exponential distribution:
q(x)=12e−x/2,x≥0,λ=12 per time unit,h(q)=1+ln⁡2≈1.693147 nats.q(x)=\frac12e^{-x/2},\quad x\ge0, \qquad \lambda=\frac12\text{ per time unit}, \qquad h(q)=1+\ln2\approx1.693147\text{ nats}.
Here λ\lambda is the rate, the reciprocal of the mean. Natural logarithms give nats; the entropy value also depends on the time coordinate, as discussed below. The density is zero for negative waits. Conrad's Theorem 3.3 establishes the fixed-mean result.
This selection has consequences beyond matching a mean. The exponential has constant hazard rate and a memoryless remaining wait. Those are properties of the selected baseline, not additional facts established about market arrivals. A mean alone does not establish independent waits or a Poisson process. The Exponential Distribution and Memorylessness develops these implications; its preview is public and the full lesson requires membership.

Why this is the maximum

A short argument uses relative entropy, or KL divergence. For any competing density ff satisfying the constraints,

DKL(f∥q)=∫0∞f(x)ln⁡f(x)q(x) dx=−h(f)+ln⁡2+Ef[X]2=1+ln⁡2−h(f)≥0.\begin{aligned} D_{\mathrm{KL}}(f\Vert q) &=\int_0^\infty f(x)\ln\frac{f(x)}{q(x)}\,dx\\ &=-h(f)+\ln2+\frac{\mathbb{E}_f[X]}{2}\\ &=1+\ln2-h(f)\ge0. \end{aligned}
Thus h(f)≤1+ln⁡2h(f)\le1+\ln2, with equality only when f=qf=q almost everywhere. The crucial cancellation is the fixed expected value: every admissible competitor has the same cross-entropy against qq. This is the argument in Strimmer's Example 6.4.

A uniform density on [0,4][0,4] also has mean two, but its differential entropy is only ln⁡4≈1.386294\ln4\approx1.386294 nats. Its hard upper bound adds concentration. A visually flat density need not maximise entropy when the permitted domain extends indefinitely.

Hypothetical waiting-time densities with mean 2: exponential rate 0.5 has differential entropy 1.693147 nats, while uniform on 0 to 4 has 1.386294 nats. The entropy gap is 0.306853 nats in the chosen time coordinate.
Hypothetical waiting-time densities with mean 2: exponential rate 0.5 has differential entropy 1.693147 nats, while uniform on 0 to 4 has 1.386294 nats. The entropy gap is 0.306853 nats in the chosen time coordinate.
Original analytic comparison. Both densities satisfy the nonnegative support and mean-two requirements. The uniform distribution additionally places all mass below four time units; entropy is compared using the same coordinate and base measure.

When equal probabilities are the answer

On a fixed set of three possible outcomes, use counting measure and impose only nonnegative probabilities and normalisation. Maximising Shannon entropy gives
p1=p2=p3=13,H(p)=−∑i=13piln⁡pi=ln⁡3≈1.098612 nats.p_1=p_2=p_3=\frac13, \qquad H(p)=-\sum_{i=1}^3p_i\ln p_i=\ln3\approx1.098612\text{ nats}.
This is the finite-outcome case of the principle, as shown in Conrad's Theorem 3.1. It does not establish that three real market scenarios occur equally often. Unequal scenario information would change the constraints or reference. There is also no uniform probability density over the entire nonnegative half-line: a positive constant cannot integrate to one there.

Units and reference choices survive the optimisation

Differential entropy uses numerical density values in specified coordinates. For Y=aXY=aX, a>0a>0, changing variables gives

h(Y)=h(X)+ln⁡a.h(Y)=h(X)+\ln a.
If the example's time unit is minutes, expressing the same wait in seconds makes its mean 120 and its entropy 1+ln⁡120≈5.7874921+\ln120\approx5.787492 nats. The physical uncertainty has not increased. This is a coordinate effect; differential entropy can also be negative. Strimmer's discussion of continuous entropy explains this lack of invariance.

A constant rescaling adds the same entropy offset to every candidate, so it preserves this optimiser when the constraints are transformed consistently. A nonlinear change of variable generally adds a distribution-dependent Jacobian term; maximising differential entropy afresh in the new coordinate can select a different law.

With a specified reference probability distribution RR, a related formulation instead minimises DKL(P∥R)D_{\mathrm{KL}}(P\Vert R) subject to constraints. KL is invariant when both distributions are transformed together. Choosing RR still requires justification. Geyer's §8 states this reference-based formulation and its connection to exponential families.

Maximum entropy therefore does not deliver a universally neutral prior, prove the data distribution, or guarantee conservative tail-risk estimates. Its defensible output is a distribution tied to stated choices. If observations reveal clustering, a finite deadline or additional moment information, those features require an updated model. More generally, constraints can be inconsistent or fail to admit an entropy maximiser; the principle alone does not guarantee existence.

Sources

Last updated
  • maximum entropy
  • distribution selection
  • moment constraints
  • reference measure
  • waiting times
Keep connecting the ideas
← Back to the glossaryBack to top ↑