Approximate Bayesian computation: Definition & Examples

Approximate Bayesian computation (ABC) is a family of methods that estimates a parameter's posterior distribution by simulating data and retaining parameter values whose simulated outputs resemble the observations. It is useful when a model is feasible to simulate but its likelihood is too expensive or unavailable to evaluate. A quant might face this problem in a market simulator with many interacting agents: generating an artificial trading session can be easier than calculating the probability of the observed session.

ABC still requires a probabilistic model, a prior and an observation mechanism. “Likelihood-free” describes the calculation it avoids; the simulator supplies the model's likelihood implicitly. Marin and coauthors develop this distinction through exact and approximate rejection samplers.

Replace a likelihood evaluation with a simulation

For the simplest rejection sampling version, let

y

denote the observed data,

S(y)

a chosen summary,

d

a distance and

\varepsilon

a tolerance. Repeat these steps independently:

Draw a proposed parameter $\theta$ from its prior $\pi(\theta)$ .
Simulate a dataset $Y^*$ under the model at that $\theta$ , using the same observation design as the real data.
Retain $\theta$ if $d(S(Y^*),S(y))\le\varepsilon$ ; otherwise discard it.

The retained parameters, rather than the simulated datasets themselves, form the sample used for Bayesian inference. Their density is proportional to

\pi_\varepsilon(\theta\mid S(y)) \propto \pi(\theta)\, \mathbb{P}_\theta\!\left(d(S(Y^*),S(y))\le\varepsilon\right).

Thus parameters that more often generate acceptable outputs receive more posterior weight. The original Beaumont–Zhang–Balding paper describes this simulate-and-compare algorithm and develops weighting and regression adjustments to it. Here we keep the unadjusted rejection rule so its target is visible.

A case where matching is exact

Consider four independent Bernoulli trials with a common unknown success probability

p

. Observe three successes and put a

\operatorname{Uniform}(0,1)

prior on

p

. This is a fictional probability experiment, not an estimate from financial data.

Use the success count $S$ as the summary. Given a proposed $p$ , simulate $S^*\sim\operatorname{Binomial}(4,p)$ . With $d(a,b)=|a-b|$ and $\varepsilon=0$ , accept exactly when $S^*=3$ . The probability of accepting this proposal is

\mathbb{P}(S^*=3\mid p)=\binom{4}{3}p^3(1-p)=4p^3(1-p).

Averaging over the prior gives the overall acceptance probability:

\mathbb{P}(\text{accept})=\int_0^1 4p^3(1-p)\,dp=\frac15.

After normalising, the accepted values have density

20p^3(1-p)

: precisely the beta distribution

\operatorname{Beta}(4,2)

. Its summaries are

\mathbb{E}[p\mid S=3]=\frac23, \qquad \operatorname{Var}(p\mid S=3)=\frac{2}{63}, \qquad \mathbb{P}(p>0.5\mid S=3)=\frac{13}{16}=81.25\%.

These are properties of the exact accepted distribution. Estimates from a finite simulation run will fluctuate around them. In this small model, analytic conjugacy is easier than ABC; the example lets us check the simulation logic against a known answer.

Count matching and sequence matching have different costs. The count event includes four possible orderings of three successes and one failure. Matching one specific observed ordering instead has probability

p^3(1-p)

at a proposed

p

, and prior-average acceptance

1/20=5\%

. Both produce the same posterior for

p

: the extra factor of four is constant in

p

and cancels in normalisation.

That equivalence rests on the assumed independent common- $p$ trials. Their count is a sufficient statistic; the ordering adds no information about $p$ . A model in which success probabilities change after previous outcomes could make ordering informative.

A wider tolerance changes the question

Keep the same summary but set $\varepsilon=1$ . Now counts 2, 3 and 4 all pass. Under this prior each possible count has prior-predictive probability $1/5$ , so acceptance rises to $3/5=60\%$ . The accepted distribution becomes

\frac13\operatorname{Beta}(3,3) +\frac13\operatorname{Beta}(4,2) +\frac13\operatorname{Beta}(5,1).

This mixture still has mean $2/3$ , but its variance is $1/21$ rather than $2/63$ . Agreement of posterior means alone would miss the change in uncertainty.

Two exact curve panels for a fictional four-trial Bernoulli model. Count tolerance zero accepts 20 percent of proposals and yields Beta(4,2); tolerance one accepts 60 percent and yields a three-beta mixture. Both posterior means are two-thirds, but variances are 2/63 and 1/21.

Exact curves for the toy model, not simulation estimates. Left: acceptance probability at each proposed $p$ . Right: the resulting normalised densities; the dotted horizontal line is the uniform prior.

Three different sources of approximation

Tolerance: accepting nearby summaries replaces the probability of the observed summary with the probability of an acceptance region. For a fixed summary and distance, tightening that region cannot increase the acceptance probability. It usually increases the work needed for a given number of retained draws.

Information loss: a summary that is not sufficient discards information before any tolerance is applied. Under suitable regularity conditions, shrinking the tolerance targets the posterior conditional on the summary; it need not recover the full-data posterior. Fearnhead and Prangle distinguish the ABC approximation from Monte Carlo error and study how to construct useful summaries.

Simulation error: a finite accepted sample only estimates the chosen ABC target. More independent proposals can reduce this error, but cannot repair a poor summary or an overly permissive tolerance. In a practical simulator, compare results across tolerances and summaries, and check whether posterior-generated data reproduce relevant features omitted from the matching rule. A good match on selected features does not validate every part of the model.

Exact matching is possible in the discrete count example. If the simulated summary has a continuous, atom-free distribution, matching a fixed observed value exactly has probability zero. Continuous data alone do not establish this: rounding, discrete summaries or probability atoms can permit positive-probability matches. Small positive tolerances avoid zero matching probability in many continuous-summary problems, but acceptance can remain extremely low, particularly with many summary coordinates. Scaling those coordinates also changes the effective comparison: a component measured in thousands can dominate one measured in decimals unless the distance accounts for their units.

For the analytic benchmark, Conjugate Prior Families works through beta–binomial updating. For the compression issue, Sufficient Statistics and the Neyman–Fisher Factorization Theorem shows why a Bernoulli count preserves information. Both have public lesson previews; full lessons require membership.

Practice: can zero tolerance still lose information?

Observe two successes in two independent Bernoulli trials with unknown common $p$ and a uniform prior. Suppose an ABC algorithm keeps only the summary “at least one success”, coded as 1 when true and 0 otherwise. It accepts an exact summary match with zero tolerance. Does it recover the full-data posterior $\operatorname{Beta}(3,1)$ ?

Answer. No. The observed summary is 1, which also accepts simulated datasets with just one success. Its acceptance probability at

p

1-(1-p)^2=2p-p^2

. Since its prior-average acceptance is

2/3

, the accepted density is

3p-\tfrac32p^2

[0,1]

, with mean

5/8

. The full-data posterior has mean

3/4

. Zero tolerance removes the discrepancy in this summary; it cannot restore the success count that the summary discarded.

Sources

Marin, Pudlo, Robert and Ryder, Approximate Bayesian Computational Methods, Sections 2–3: exact rejection, the ABC target and calibration.
Beaumont, Zhang and Balding (2002), Approximate Bayesian Computation in Population Genetics, introduction and Methods: rejection ABC, scaling, weighting and regression adjustment.
Fearnhead and Prangle, Constructing Summary Statistics for Approximate Bayesian Computation: Semi-automatic ABC, Sections 1.1–1.2: summary-based targets and the distinction between approximation and Monte Carlo error.

Replace a likelihood evaluation with a simulation

A case where matching is exact

A wider tolerance changes the question

Three different sources of approximation

Practice: can zero tolerance still lose information?

Sources

Related terms

Bernoulli distribution

Beta distribution

Likelihood

Rejection sampling