Dirichlet process mixture: Definition & Examples

A Dirichlet process mixture (DPM) is a Bayesian mixture model that places a Dirichlet process prior on the distribution of component parameters. It allows an unbounded supply of components while any finite dataset occupies only finitely many. A quant can use this flexibility to model a return-residual density without fixing the number of Gaussian components in advance. The resulting clusters describe statistical variation; they do not establish economic regimes.

A random mixing measure, then observations

The hierarchy is

G\sim\operatorname{DP}(\alpha,H),\qquad \theta_i\mid G\sim G,\qquad x_i\mid\theta_i\sim f(x_i\mid\theta_i).

Here

H

is a probability distribution over component parameters,

\alpha>0

is the concentration parameter, and

f

is the observation kernel. For a Gaussian kernel,

\theta_i

could contain a mean and variance. Conditional on

G

, the parameter draws are independent; conditional on their parameters, the observations are independent. Neal's model formulation makes these layers explicit.

The Dirichlet process is a distribution over probability measures. It extends the finite-dimensional Dirichlet distribution: the masses assigned by

G

to any finite measurable partition have a Dirichlet law. It is not a probability distribution for a single observation.

A draw $G$ is atomic almost surely. Writing its masses as $\pi_k$ at parameters $\theta_k^*$ gives the mixture density

p(x\mid G)=\int f(x\mid\theta)\,G(d\theta) =\sum_{k=1}^{\infty}\pi_k f(x\mid\theta_k^*).

Atomic mixing does not force atomic observations: Gaussian kernels produce an absolutely continuous observation distribution. Repeated

\theta_i

values identify a shared component, not repeated

x_i

values. Teh's account explains this smoothing distinction.

What the restaurant rule actually says

Assume $H$ is nonatomic, so a newly drawn component parameter does not coincide with an existing one. After integrating out $G$ , let $z_{1:n}$ denote existing latent assignments and let $n_k$ be cluster $k$ 's occupancy. Before conditioning on the next observed value,

\mathbb{P}(z_{n+1}=k\mid z_{1:n})=\frac{n_k}{n+\alpha},\qquad \mathbb{P}(z_{n+1}=\text{new}\mid z_{1:n})=\frac{\alpha}{n+\alpha}.

This is the Chinese restaurant process rule. With counts 3 and 1 and

\alpha=2

, the three weights are

3/6=1/2

1/6

, and

2/6=1/3

. These are conditional prior allocation probabilities, not posterior assignments after seeing

x_{n+1}

For the latter, Bayes' theorem also uses the component predictive densities. With hyperparameters fixed and the existing partition given, write

m_k(x)

for the predictive density from cluster

k

and

m_0(x)=\int f(x\mid\theta)H(d\theta)

for a fresh component. The unnormalized weights become

n_k m_k(x)\quad\text{and}\quad\alpha m_0(x).

Normalize across all existing clusters and the new-cluster option. This is the same likelihood weighting underlying Neal's collapsed allocation updates; a full posterior also averages over uncertainty in the existing partition.

For an original arithmetic illustration, stipulate predictive densities at one new

x

0.1

0.8

, and

0.2

in the same inverse-data units. They are density evaluations, not three probabilities required to sum to one. Multiplying the prior weights gives

0.05

0.1333\ldots

, and

0.0667\ldots

, whose sum is

0.25

. The normalized allocation is 20%, 53.33%, and 26.67%. The smaller existing cluster now leads because it explains this observation better.

Synthetic allocation example with existing cluster counts 3 and 1 and concentration 2. Before the new observation, allocation probabilities are 50%, 16.7%, and 33.3% for cluster 1, cluster 2, and a new cluster. Stipulated same-unit predictive densities 0.1, 0.8, and 0.2 change the conditional probabilities to 20%, 53.3%, and 26.7%.

Original conditional calculation, not fitted market data. The solid bars use the restaurant prior; outlined bars also condition on the new observation while holding the existing partition fixed.

Flexible density does not mean a discovered true cluster count

The concentration parameter affects the prior number of occupied clusters, rather than fixing it. Under this restaurant prior,

\mathbb{E}[K_n]=\sum_{i=1}^{n}\frac{\alpha}{\alpha+i-1}.

For $n=4$ and $\alpha=2$ , this prior expectation is $77/30\approx2.567$ , even though the particular partition above has two clusters. Cluster labels themselves are arbitrary.

A useful density fit does not establish consistent estimation of a finite data-generating component count. Miller and Harrison prove inconsistency of the posterior occupied-cluster count for broad DPM families with fixed concentration, including Gaussian mixtures with the stated conjugate base measures. This is a specific limitation of count inference, not a claim that DPM density estimation always fails.

The basic model is also exchangeable: it does not model persistence between consecutive trading dates. A component with high variance need not be a persistent stress state. Kernel choice,

H

, and concentration assumptions still shape tail predictions; an unlimited component supply removes none of those modeling decisions.

For the finite-mixture comparison, Mixture of Gaussians and the EM Algorithm derives likelihood-based soft assignments with a fixed component count. Public preview; the full lesson requires membership.

Sources

Yee Whye Teh (2010), Dirichlet Process: finite-partition definition, Chinese restaurant process, expected cluster count, and mixture hierarchy.
Radford M. Neal (2000), Markov Chain Sampling Methods for Dirichlet Process Mixture Models, §§2–3: model layers and allocation updates with likelihood factors.
Jeffrey W. Miller and Matthew T. Harrison (2014), Inconsistency of Pitman–Yor Process Mixtures for the Number of Components, §1.1 and Corollary 1: conditions and limits of occupied-cluster count inference.

A random mixing measure, then observations

What the restaurant rule actually says

Flexible density does not mean a discovered true cluster count

Sources

Related terms

Bayes' theorem

Exchangeability and de Finetti theorem