Variational inference and evidence lower bound: Definition & Examples

Variational inference (VI) approximates a difficult probability distribution by choosing a tractable family of distributions and optimising a member of that family. In Bayesian modelling, the target is usually a posterior distribution. For a quant fitting a latent-factor or regime model, VI can make repeated posterior fitting affordable—but the reported uncertainty depends on which shapes the approximation permits.

The evidence lower bound (ELBO) is the objective used in the standard reverse-KL formulation of VI. It makes posterior approximation possible without first evaluating the model evidence. A high ELBO is useful evidence of fit within a specified model and variational family; it does not certify that uncertainty is accurate.

An objective that avoids the evidence integral

Let $x$ be observed data and $z$ the unknown quantities, including any parameters treated as random. A joint model $p(x,z)$ gives

p(z\mid x)=\frac{p(x,z)}{p(x)}, \qquad p(x)=\int p(x,z)\,dz.

The integral may be difficult even when the joint density is easy to evaluate. Choose a normalised approximation $q_\lambda(z)$ , with variational parameters $\lambda$ , and define

\mathcal L(\lambda) =\mathbb E_{q_\lambda}[\log p(x,z)] -\mathbb E_{q_\lambda}[\log q_\lambda(z)].

For fixed $x$ and model, the exact identity is

\log p(x)=\mathcal L(\lambda) +\operatorname{KL}\!\left(q_\lambda(z)\,\|\,p(z\mid x)\right).

Here KL divergence is $\mathbb E_q[\log q-\log p]$ . The identity follows by substituting $\log p(z\mid x)=\log p(x,z)-\log p(x)$ into that expectation. Assume $0<p(x)<\infty$ , that the posterior is positive wherever $q$ has positive density, and that the expectations are finite. Nonnegativity of KL then gives $\mathcal L(\lambda)\leq\log p(x)$ .

The direction matters: the expectation is under the approximation, so the objective minimises

\operatorname{KL}(q\|p)

, not

\operatorname{KL}(p\|q)

. The latter generally requires expectations under the posterior we cannot yet compute. The ELBO uses the joint model and

q

instead. Jordan and coauthors' variational construction, Section 6, connects this lower bound to KL minimisation.

Maximising the ELBO therefore reduces the posterior KL for the same fixed model. Equality with log evidence occurs only when

q

equals the posterior almost everywhere. A restricted family can leave a gap even after perfect optimisation. If model parameters are changed too, log evidence also changes; an ELBO increase alone no longer measures a reduction in posterior KL.

A normal target: when the family can be exact

Take a synthetic posterior $p(z\mid x)=\mathcal N(1,4)$ and a Gaussian approximation $q(z)=\mathcal N(m,s^2)$ . The second normal parameter denotes variance. Direct integration gives

\operatorname{KL}(q\|p) =\frac12\left[\log\frac{4}{s^2} +\frac{s^2+(m-1)^2}{4}-1\right].

Starting from $q=\mathcal N(0,1)$ produces KL $0.443147$ in natural-log units. Its ELBO is exactly $0.443147$ below log evidence. Allowing both mean and variance to move gives $m=1$ , $s=2$ and KL zero: the chosen family contains the target. Restricting the mean or variance would remove that guarantee.

The target normal density has mean 1 and variance 4; the initial approximation has mean 0 and variance 1. Its KL gap is 0.443147, falling to zero when both parameters match the target.

Figure 1. A synthetic target that the Gaussian family can reproduce exactly. The solid curve is both the posterior and the optimal approximation; the dashed curve is the initial approximation.

Mean-field can shrink uncertainty even at the optimum

A common simplification is mean-field factorisation,

q(z)=\prod_{j=1}^k q_j(z_j).

The factors can be flexible, but their product imposes independence between the chosen coordinates or blocks. This is a computational assumption about the approximate posterior, not a claim that the original model or its posterior has independent components. Block factorizations can retain dependence within a block.

Consider a synthetic two-dimensional Gaussian posterior with zero means and covariance

\Sigma=\begin{pmatrix}1&0.8\\0.8&1\end{pmatrix}.

Fit independent Gaussian factors with means zero and variances $v_1,v_2$ . Zero means are also optimal when the means are free. The KL objective becomes

\operatorname{KL}(q\|p) =\frac12\left[ \frac{v_1+v_2}{0.36}-2 +\log\frac{0.36}{v_1v_2} \right].

Differentiating with respect to each variance gives

\tfrac12(1/0.36-1/v_j)

, so the optimum has

v_1=v_2=0.36

. The target marginal variances are 1, not

0.36

. Thus each approximate marginal has standard deviation

0.6

rather than

1

, while the remaining KL is

-\frac12\log(0.36)=0.510826.

A correlated Gaussian target has an elongated contour with correlation 0.8 and marginal variances 1. Its optimal independent approximation has a circular contour, marginal variances 0.36, standard deviations 0.6 and residual KL 0.510826.

Figure 2. Original contours and marginal densities for the synthetic example. Both contours use Mahalanobis radius 2; the right panel shows the loss of marginal spread as well as the loss of dependence.

If these coordinates represented standardised factor coefficients, the posterior variance of their sum would be $1+1+2(0.8)=3.6$ . Under the fitted approximation it would be $0.36+0.36=0.72$ . Accurate means would conceal a substantial error in uncertainty about their combined effect.

This example illustrates a known mean-field limitation discussed by Blei, Kucukelbir and McAuliffe, Section 2.3. Putting approximate mass where the target density is very small is expensive under reverse KL. A restricted approximation may consequently narrow or favour one mode. “VI always finds one mode” is not a theorem: behaviour depends on the target, family and optimiser. Here the target has only one mode, yet the variance error is substantial. A full-covariance Gaussian family would contain this target and remove this particular approximation error.

Fitting at scale—and deciding what to trust

For some models, coordinate ascent updates each factor while holding the others fixed:

q_j^*(z_j)\propto \exp\!\left(\mathbb E_{q_{-j}}[\log p(x,z)]\right).

Here

q_{-j}

is the product of the other factors. Such expectations are not always tractable. Gradient methods provide another route, using analytic or Monte Carlo estimates of ELBO gradients. Ranganath, Gerrish and Blei’s black-box method, for example, estimates gradients using samples from the variational distribution.

Stochastic variational inference uses subsampled data to form cheaper noisy gradients. If the model's ELBO separates into a global term and

N

observation contributions, a uniform minibatch of size

B

estimates their full sum by scaling the minibatch sum by

N/B

. The global prior and entropy terms are not multiplied by that factor. Hoffman and coauthors, Sections 2.4–2.5, develop natural-gradient updates and minibatches for a class of hierarchical models. Appropriate step sizes and gradient estimators matter; noisy individual updates need not increase the ELBO. For financial time series, preserve the model's serial dependence rather than treating arbitrary rows as independent observations.

There are two different shortcomings to investigate: optimisation error, from failing to find the best member of a family, and approximation error, from the family's inability to represent the target. More iterations address the first; they cannot restore a correlation prohibited by the second. A richer family may help but can make fitting harder.

Markov chain Monte Carlo instead constructs samples with the posterior as their stationary target. Under suitable conditions it can approach exact posterior expectations as the run grows, without committing to a fixed variational shape. Finite runs still face mixing and Monte Carlo error. VI can be attractive for repeated or large-scale fits, but neither method has a universal speed or accuracy advantage.

For a risk-sensitive application, compare posterior spreads, tail probabilities and relevant functions such as the factor sum above—not just fitted means or a stable optimisation trace. Multiple starts probe optimisation sensitivity. Comparisons with a richer family or well-diagnosed MCMC probe approximation sensitivity.

For a longer treatment of the factor update, see SelfMadeQuant's Variational Inference and a Frontier Pointer to Deep Generative Models. The link opens a public lesson preview; the complete lesson requires membership.

Practice: is matching the mean enough?

A zero-mean Gaussian target has unit marginal variances and correlation $0.6$ . You fit the best independent Gaussian approximation under reverse KL. What marginal variance does it select, and can further optimisation recover the target's correlation?

Answer. The same calculation gives

v_j=1-0.6^2=0.64

and residual KL

-\tfrac12\log(0.64)=0.223144

. Further optimisation cannot create dependence inside a family that forces independence. Changing the family is necessary to represent the target correlation.

Sources

Jordan, Ghahramani, Jaakkola and Saul (1999), An Introduction to Variational Methods for Graphical Models, Section 6: approximate families, KL direction and the lower bound.
Hoffman, Blei, Wang and Paisley (2013), Stochastic Variational Inference, Sections 2.4–2.5: stochastic natural gradients, step sizes and minibatches.
Ranganath, Gerrish and Blei (2014), Black Box Variational Inference, introduction: Monte Carlo gradients and the limits of closed-form coordinate ascent.
Blei, Kucukelbir and McAuliffe (2017), Variational Inference: A Review for Statisticians, Sections 1 and 2.2–2.4: ELBO identity, Gaussian mean-field geometry, coordinate ascent and comparison with MCMC. The linked author manuscript was revised in 2018.