Skip to content

Horseshoe prior

A continuous global-local shrinkage prior: strong concentration near zero with heavy tails, and a careful distinction between conditional Gaussian updates and full posterior inference.

The horseshoe prior is a continuous global-local shrinkage prior that concentrates strongly near zero while retaining heavy tails. In a large factor model, it lets a modeller express the belief that most exposures are small without forcing every potentially large exposure to face the same tight Gaussian prior.

One global scale, many escape routes

A normal-means version is

yj∣θj∼N(θj,1),θj∣λj,τ∼N(0,τ2λj2),λj∼iidC+(0,1).\begin{aligned} y_j\mid\theta_j &\sim \mathcal{N}(\theta_j,1),\\ \theta_j\mid\lambda_j,\tau &\sim \mathcal{N}(0,\tau^2\lambda_j^2),\\ \lambda_j &\stackrel{\text{iid}}{\sim} C^+(0,1). \end{aligned}
Here C+(0,1)C^+(0,1) is the standard half-Cauchy distribution, with density 2/[π(1+λ2)]2/[\pi(1+\lambda^2)] for λ>0\lambda>0. The half-Cauchy is on the local standard-deviation scale λj\lambda_j, not on its square. The shared τ>0\tau>0 pulls coefficients toward zero; a large local scale allows an individual coefficient to escape. Coefficients are independent conditional on the shared scale, not generally after integrating it out. Carvalho, Polson and Scott (2009) introduce this hierarchy.
A full specification also needs a choice for τ\tau. One conventional hierarchy adds τ∼C+(0,τ0)\tau\sim C^+(0,\tau_0), with scale τ0>0\tau_0>0; fixing τ\tau instead defines a conditional version. The global scale is consequential: its prior should reflect noise, predictor scales and plausible sparsity rather than being treated as an innocuous default. Piironen and Vehtari's hyperprior analysis explains this sensitivity.

What holding the scales fixed actually buys

Write a=τ2λ2a=\tau^2\lambda^2 for one coefficient's conditional prior variance. With unit observation-noise variance, adding Gaussian precisions gives

v=(1+1a)−1=a1+a,m=vy.v=\left(1+\frac1a\right)^{-1}=\frac{a}{1+a}, \qquad m=vy.

Equivalently, set κ=1/(1+a)\kappa=1/(1+a). Then

θ∣y,λ,τ∼N((1−κ)y, 1−κ).\theta\mid y,\lambda,\tau\sim\mathcal{N}\big((1-\kappa)y,\,1-\kappa\big).
Thus κ\kappa is the fraction of the observation removed by shrinkage, while 1−κ1-\kappa is the fraction retained. This is ordinary Normal–Normal conjugacy conditional on the scales.

For a fictional standardized exposure observation y=3y=3, fix τ=0.5\tau=0.5 and compare two supplied local scales:

Supplied λ\lambdaPrior variance aaShrinkage κ\kappaConditional posterior meanConditional posterior variance
0.20.20.010.01100/101≈0.9901100/101\approx0.99013/101≈0.02973/101\approx0.02971/101≈0.0099011/101\approx0.009901
44440.20.22.42.40.80.8
The same observation nearly disappears under the small local scale and mostly survives under the large one. These are two conditional Gaussian updates, not two fitted horseshoe estimates or an integrated horseshoe posterior mean.
Illustrative normal-means model with observation 3, noise variance 1 and fixed global scale 0.5. A supplied local scale of 0.2 gives conditional posterior mean 0.0297; local scale 4 gives 2.4. The curve shows conditional means, not integrated horseshoe estimates.
Illustrative normal-means model with observation 3, noise variance 1 and fixed global scale 0.5. A supplied local scale of 0.2 gives conditional posterior mean 0.0297; local scale 4 gives 2.4. The curve shows conditional means, not integrated horseshoe estimates.
Original deterministic illustration. The local scale changes how much of the same observation survives conditional Gaussian updating; the scales shown are supplied, not fitted.

With fixed τ\tau but uncertain λ\lambda, the actual horseshoe mean is

E[θ∣y,τ]=(1−E[κ∣y,τ])y.\mathbb{E}[\theta\mid y,\tau] =\big(1-\mathbb{E}[\kappa\mid y,\tau]\big)y.
The expectation is under the posterior of the scale, not its prior. If τ\tau is also uncertain, it must be integrated out as well. A convenient conditional formula does not eliminate that inference problem.

Two peaks, no on–off switch

For unit noise variance and τ=1\tau=1, transforming the half-Cauchy local scale gives

p(κ)=1πκ(1−κ),0<κ<1.p(\kappa)=\frac{1}{\pi\sqrt{\kappa(1-\kappa)}},\qquad 0<\kappa<1.
This is a Beta⁡(1/2,1/2)\operatorname{Beta}(1/2,1/2) density: high near both little shrinkage and near-total shrinkage, hence “horseshoe.” It is a prior density, not evidence that a given fitted coefficient is a signal. Its symmetry is specific to these unit scales; changing τ\tau changes the balance. Piironen and Vehtari, §2 give the general profile.
With global scale 1 and noise variance 1, the shrinkage weight has a continuous Beta one-half, one-half prior density. It is unbounded near 0 and 1, with no point mass at either endpoint. The plotted range is 0.005 to 0.995.
With global scale 1 and noise variance 1, the shrinkage weight has a continuous Beta one-half, one-half prior density. It is unbounded near 0 and 1, with no point mass at either endpoint. The plotted range is 0.005 to 0.995.
Original plot of the prior shrinkage profile in the unit-scale normal-means model. The endpoint poles explain the name; they do not represent discrete inclusion decisions.
After integrating out λ\lambda at fixed positive τ\tau, the coefficient density has an integrable pole at zero and Cauchy-like tails. An infinite density at a point still gives that point zero probability. Unlike a spike-and-slab prior with a point-mass spike, the horseshoe does not automatically assign positive posterior probability to θj=0\theta_j=0. Reporting an exactly sparse model needs an additional selection rule. See Carvalho, Polson and Scott, §§1–2.

When escaping shrinkage is too easy

The regularized horseshoe is a separate extension. It replaces the local variance factor by
λ~j2=c2λj2c2+τ2λj2,θj∣λj,τ,c∼N(0,τ2λ~j2).\widetilde{\lambda}_j^2 =\frac{c^2\lambda_j^2}{c^2+\tau^2\lambda_j^2}, \qquad \theta_j\mid\lambda_j,\tau,c \sim\mathcal{N}(0,\tau^2\widetilde{\lambda}_j^2).
For fixed c>0c>0, its conditional prior variance approaches c2c^2 as λj\lambda_j grows. That supplies regularization even for very large coefficients, which can matter when the likelihood barely constrains them, as with separated logistic-regression data. Sending c→∞c\to\infty recovers the original horseshoe. Piironen and Vehtari, §2.3 develop this extension; its finite-slab behavior should not be attributed to the original prior.
The scalar calculation above also does not make correlated regression coefficients independent. A Bayesian linear regression uses the full design matrix in its conditional covariance; similar predictors can remain difficult to distinguish even with shrinkage.
For the algebra underneath the fixed-scale example, Conjugate Prior Families derives the Normal–Normal precision update (full lesson requires membership; public preview available).

Sources

Last updated
  • bayesian inference
  • global local shrinkage
  • sparsity
  • half cauchy
Keep connecting the ideas
← Back to the glossaryBack to top ↑