Approximate rare stress episodes using their clump rate and occupied duration, while separating threshold crossings, time occupancy and window probabilities.
Poisson clumping heuristic
The Poisson clumping heuristic approximates rare visits to an extreme region as short clusters whose arrival times behave approximately like a Poisson process. It lets a quant turn “how much time does a system spend beyond a threshold?” into “how often does a new stress episode begin?” Dependence within an episode is retained; approximate independence is sought between well-separated episodes.
A trading-system latency monitor makes the distinction concrete. Ten alarms during one overloaded spell may represent one operational incident, rather than ten independent opportunities for overload to begin. The heuristic models the timing of those incidents. It does not estimate the financial loss attached to them.
The clock inside a clump
Let Xt be a stationary monitored process, fix a high threshold u, and write
p=P(Xt>u).
Here p is the probability of being above the threshold at a fixed time. Group nearby exceedances into clumps, let λ be their start rate, and let C be the total time above u within a typical clump, selected by its start. A brief dip below the threshold can lie inside the same clump, so C need not equal the time from the clump's first crossing to its last exit.
The central rare-event balance is
p≈λE[C],λ≈E[C]p.
The units explain the balance: clumps per minute multiplied by occupied minutes per clump gives a fraction of the clock. Aldous's original treatment, §§A4 and A8, uses this local clump calculation to approximate a global rare-event waiting problem.
Counting every upward crossing of u generally gives a different rate: a single clump may cross repeatedly. Equally, selecting an arbitrary above-threshold instant overrepresents clumps with more occupied time. Neither count should silently replace start-selected clumps when estimating λ or E[C].
Six per cent of the clock, yet a seventy per cent hourly chance
For an original hypothetical latency monitor, stipulate a clump rate of 0.02 per minute and a mean occupied duration of 3 minutes per clump. These inputs are an illustration, not measured parameters or a claim that real latency obeys the heuristic. Assume rare, well-separated episodes and an approximately constant Poisson start rate over the window of interest.
The occupied fraction is approximately
p≈0.02×3=0.06=6%.
Now count new clump starts in the next T=60 minutes. If NT is approximately Poisson with mean λT, then
There is no contradiction. 6% concerns a single observation time; 69.88% concerns whether at least one episode begins during an hour. And 1.2 is an expected count, not a probability or a possible realised integer count.
Top: a separate schematic groups four threshold upcrossings into two clumps, each with three minutes above threshold but spans of four and five minutes. Bottom: the stipulated Poisson-start approximation at 0.02 per minute gives a 69.88% probability of at least one new clump start in 60 minutes; the rare-time occupancy approximation is 6%.
Original grouping schematic and analytic approximation. The upper path explains the clump definition; it is neither observed data nor a draw from the lower model. The lower curve counts new starts, rather than all threshold crossings or incidents already in progress.
Under the same Poisson-start approximation, the wait W to a new start has an exponential distribution:
P(W>t)≈e−0.02t,E[W]≈50minutes.
An alarm already active at time zero is a separate boundary condition. A first-passage time to Xt>u is zero if the process already exceeds u. Starting just below u may also produce a rapid recrossing belonging to the current episode. The 50-minute approximation is useful from a typical state away from the rare region, once those local effects are negligible.
For the exact count and waiting-time machinery underlying this approximation, Defining the Poisson Process: Three Equivalent Constructions connects independent exponential clocks to Poisson counts. The link provides a public preview; the full lesson requires membership.
Where the approximation can mislead
Rarity alone does not create a Poisson clock. Persistent regimes or dependence across episodes can invalidate the exponential waiting approximation. The clump rule also needs a useful separation of time scales: long enough to group local returns, short relative to the gaps between independent episodes. If changing that rule radically changes the result, the reported start rate needs further justification.
Overlap changes what is being counted. In a separate, exact toy model, place intervals of fixed length 3 minutes at the points of a stationary Poisson process of rate 0.02 per minute. A fixed instant is covered precisely when at least one start lies in the preceding 3 minutes. Its coverage probability is therefore
1−e−0.02×3≈5.8235%,
rather than exactly 6%. The product counts interval time with multiplicity. Merging overlapping intervals changes the episode rate and duration; the original start rate is no longer the rate of merged episodes. This coverage formula is exact for this particular toy model, not a universal correction for dependent threshold exceedances. Conversely, for genuinely disjoint episodes with every occupied instant assigned once, stationary time accounting can give p=λE[C] exactly; that equality alone does not prove Poisson arrivals.
The practical modelling question is therefore whether the chosen clumps isolate local persistence and leave sufficiently separated arrivals. The heuristic supplies an approximation after that choice; it does not validate the choice or provide an error bound by itself.
Sources
David Aldous, Probability Approximations via the Poisson Clumping Heuristic (1989), author-hosted full text: §§A2–A4, pp. 2–6, give Poisson clump coverage and the sparse approximation; §A6, pp. 8–9, distinguishes ordinary and size-biased clump sampling; §A8, pp. 11–12, distinguishes occupied length from span; §§B1–B2, pp. 23–24, state the hitting-time heuristic, its initial-state qualifications and its lack of universal validity.
