Bayesian linear regression: Definition & Examples

Bayesian linear regression puts a prior distribution on the coefficients of a linear regression and updates it with observed data to obtain a posterior distribution. A quant estimating a factor exposure can then carry uncertainty about that exposure into a conditional prediction, instead of treating one fitted slope as certain.

The crucial distinction is between uncertainty about a coefficient, uncertainty about the mean response, and uncertainty about a new observation. Those three distributions can have very different variances.

Specify the model before updating it

For a design matrix $X$ with one observation per row, consider

\mathbf y=X\boldsymbol\beta+\boldsymbol\varepsilon, \qquad \boldsymbol\varepsilon\sim\mathcal N(\mathbf 0,\sigma^2 I), \qquad \boldsymbol\beta\sim\mathcal N(\mathbf m_0,V_0).

Here inputs are conditioned on, noise is independent across observations and independent of the coefficients,

\sigma^2>0

is known, and

V_0

is positive definite. An intercept, if wanted, needs its own column of ones and a specified prior. “Linear” means linear in the coefficients; predetermined nonlinear transformations of inputs can also appear as columns of

X

The prior expresses assumptions before these observations. Multiplying its density by the likelihood and normalising gives a Gaussian posterior:

\begin{aligned} V_n&=\left(V_0^{-1}+\frac{X^\top X}{\sigma^2}\right)^{-1},\\ \mathbf m_n&=V_n\left(V_0^{-1}\mathbf m_0+\frac{X^\top\mathbf y}{\sigma^2}\right),\\ \boldsymbol\beta\mid D&\sim\mathcal N(\mathbf m_n,V_n), \qquad D=(X,\mathbf y). \end{aligned}

Prior and data precisions, meaning inverse covariances here, add. These are the Gaussian update formulas developed in Rasmussen and Williams, §2.1.1, written with observations in rows rather than their column convention.

Two observations, three uncertainty statements

Take an original, synthetic scalar example in arbitrary units:

y_i=x_i\beta+\varepsilon_i, \qquad \beta\sim\mathcal N(0,1), \qquad \varepsilon_i\stackrel{\mathrm{iid}}{\sim}\mathcal N(0,1).

There is no intercept. Observe

(x_1,y_1)=(1,1)

and

(x_2,y_2)=(2,2)

. Although the two points lie exactly on

y=x

, the model stipulates noise variance one; two neat points do not make it zero.

The scalar update is

v_n=\frac{1}{1+1^2+2^2}=\frac16, \qquad m_n=v_n(1\cdot1+2\cdot2)=\frac56.

Thus $\beta\mid D\sim\mathcal N(5/6,1/6)$ . To see the update directly, the negative log posterior, up to constants and a factor of one half, contains

\beta^2+(1-\beta)^2+(2-2\beta)^2 =6\left(\beta-\frac56\right)^2+\frac56.

The no-intercept least-squares slope is $1$ . The posterior mean $5/6$ is pulled toward the prior mean zero: a concrete instance of shrinkage.

Now supply a new input $x_*=3$ . Define the latent response $f_*=3\beta$ and a fresh observation $y_*=3\beta+\varepsilon_*$ , where the new noise is independent and has variance one.

Quantity, conditional on the observed data	Mean	Variance
Coefficient $\beta$	$5/6$	$1/6$
Latent mean response $f_*=3\beta$	$2.5$	$9/6=1.5$
New observation $y_$ at $x_=3$	$2.5$	$1+1.5=2.5$

Coefficient variance is in squared coefficient units; the other two variances are in squared response units. Equal means do not make the latent response and the new observation interchangeable.

Synthetic no-intercept example. The coefficient prior has mean zero and variance one; its posterior has mean five sixths and variance one sixth. At input three, the latent response and new observation both have mean 2.5, with variances 1.5 and 2.5 respectively.

Original synthetic example. The upper panel compares coefficient densities; the lower panel compares response densities at a supplied input of three. The wider dashed response curve includes fresh observation noise.

Prediction averages over the coefficients

For a supplied vector $\mathbf x_*$ , integrating over the coefficient posterior gives

\begin{aligned} f_*\mid D,\mathbf x_*&\sim\mathcal N(\mathbf x_*^\top\mathbf m_n,\;\mathbf x_*^\top V_n\mathbf x_*),\\ y_*\mid D,\mathbf x_*&\sim\mathcal N(\mathbf x_*^\top\mathbf m_n,\;\sigma^2+\mathbf x_*^\top V_n\mathbf x_*). \end{aligned}

The second variance adds the noise remaining even if the coefficients were known. Plugging the posterior mean into the observation model would retain only

\sigma^2

and discard coefficient uncertainty. Hughes's Tufts notes, Parts 2–4, contrast these prediction strategies.

In the example, $x_*=3$ lies outside the observed input range. The larger latent variance reflects slope uncertainty under the stipulated straight-line model. It does not measure how wrong that model might be outside its training range. At $x_*=0$ , latent variance is exactly zero because the no-intercept specification forces the mean response to zero; a new observation still has variance one.

What the posterior leaves conditional

A zero-centred Gaussian prior produces a quadratic coefficient penalty. In the isotropic case $V_0=\tau^2I$ , the posterior mode minimises squared residuals plus $(\sigma^2/\tau^2)\lVert\boldsymbol\beta\rVert^2$ : the ridge regression connection. Matching a point estimate does not supply the whole posterior. Prior scale also depends on input units; rescaling a predictor while retaining the same numerical coefficient prior changes the model.

Known noise variance is a deliberate simplification. If variance is unknown, it needs a prior and joint updating. Under an appropriate conjugate normal–inverse-gamma specification, integrating it out produces Student-t coefficient and predictive distributions; this is not achieved by silently replacing

\sigma^2

with a residual estimate. Taboga's treatment separates the known- and unknown-variance models. Other priors need not give the same distributional form.

For financial data, serially dependent or changing-variance errors, unstable exposures and omitted predictors can invalidate the stated likelihood. A narrow posterior is conditional on that likelihood and prior. It is not evidence that those choices are correct.

Finally, an exposure regression using market and asset returns from the same period is an in-sample relationship. A forecast requires inputs available at its decision time, a specified horizon and evaluation on later unseen outcomes. If a future predictor is itself unknown, these conditional formulas do not integrate its uncertainty. Bayesian updating does not remove look-ahead bias.

For the conjugate-update foundation, read Bayesian Reasoning. For the least-squares comparison and coefficient inference, read Linear Regression. Both full lessons require membership, with public previews.

Sources

Carl Edward Rasmussen and Christopher K. I. Williams, Gaussian Processes for Machine Learning, Chapter 2, §2.1.1–2.1.2 — Gaussian coefficient posterior, latent prediction, ridge connection and fixed feature maps.
Michael C. Hughes, Tufts University, “Day01-A: Bayesian Linear Regression,” Parts 1–4 and 6 — known hyperparameters, posterior integration, observation predictive variance and hyperparameter uncertainty.
Marco Taboga, “Bayesian linear regression,” Lectures on probability theory and mathematical statistics — Gaussian regression with known variance and the conjugate unknown-variance extension.

Specify the model before updating it

Two observations, three uncertainty statements

Prediction averages over the coefficients

What the posterior leaves conditional

Sources

Related terms

Likelihood

Linear regression

Look-ahead bias

Posterior probability

Prior probability