Quadratic discriminant analysis: Definition & Examples

Quadratic discriminant analysis (QDA) is a classifier that models the feature distribution within each class as Gaussian, with a separate mean and covariance matrix for each class, then applies Bayes' rule to predict class probabilities. Its practical attraction is that classes can differ in their spread and correlations as well as their average features. A quant investigating labelled market conditions can therefore test whether unusually large moves in either direction contain information that a straight decision boundary would miss.

The Gaussian assumption applies within each labelled class. It does not say the pooled observations are Gaussian, and it does not establish that financial returns follow this model.

From a feature cloud to a class probability

Let $Y\in\{1,\ldots,K\}$ be the class and $\mathbf{x}\in\mathbb{R}^p$ the observed feature vector. QDA assumes

\mathbf{X}\mid Y=k\sim\mathcal{N}(\boldsymbol\mu_k,\Sigma_k), \qquad \mathbb{P}(Y=k)=\pi_k, \qquad \sum_{k=1}^K\pi_k=1.

Here

\pi_k>0

is the class prior,

\boldsymbol\mu_k

its mean vector and

\Sigma_k

a positive definite covariance matrix. A multivariate normal distribution gives each class a density

f_k(\mathbf{x})

. Bayes' theorem turns the prior-weighted densities into posteriors:

\mathbb{P}(Y=k\mid\mathbf{x}) =\frac{\pi_k f_k(\mathbf{x})}{\sum_{j=1}^K\pi_j f_j(\mathbf{x})}.

The denominator matters when reporting probabilities. For choosing the most likely class, it is common to all classes and can be omitted. Equivalently, maximise the discriminant score

\delta_k(\mathbf{x}) =\log\pi_k-\frac12\log\det\Sigma_k -\frac12(\mathbf{x}-\boldsymbol\mu_k)^\top \Sigma_k^{-1}(\mathbf{x}-\boldsymbol\mu_k).

The Gaussian constant

-p\log(2\pi)/2

is the same for every class. The official scikit-learn derivation gives this score and its relationship to the posterior.

Read the three terms together. The prior rewards a more common class. The squared Mahalanobis distance measures deviation using that class's shape. The log determinant accounts for the volume over which its density is spread: an expansive class has lower density at its centre. Ignoring this term would give a wide class an unfair advantage simply because many points look close in its own variance units.

To recover probabilities from the scores, normalise their exponentials:

\mathbb{P}(Y=k\mid\mathbf{x}) =\frac{\exp(\delta_k(\mathbf{x})-m)} {\sum_j\exp(\delta_j(\mathbf{x})-m)}, \qquad m=\max_j\delta_j(\mathbf{x}).

Subtracting

m

leaves the ratios unchanged and avoids exponentiating large positive numbers. Selecting the largest posterior minimises expected classification error under equal costs for all mistakes. Unequal losses require a decision based on posterior expected loss; the most probable label need not be the preferred action.

Same average, different tails

Consider a fictional population illustration, with one dimensionless market feature

x

and two externally defined classes. Supply the parameters rather than estimate them from a dataset:

X\mid Y=1\sim\mathcal{N}(0,1),\qquad X\mid Y=2\sim\mathcal{N}(0,4),\qquad \pi_1=\pi_2=\frac12.

Class 2 has twice the standard deviation of class 1. Their means coincide, so all separation comes from spread. Taking the density ratio gives the posterior log odds

\begin{aligned} \log\frac{\mathbb{P}(Y=2\mid x)}{\mathbb{P}(Y=1\mid x)} &=\log\frac{\pi_2}{\pi_1}-\frac12\log\frac{4}{1} -\frac{x^2}{2\cdot4}+\frac{x^2}{2\cdot1}\\ &=-\log 2+\frac38x^2. \end{aligned}

There are two decision boundaries, found by setting these log odds to zero:

x=\pm\sqrt{\frac{8\log2}{3}}\approx\pm1.3596.

Predict class 1 between the boundaries and class 2 outside them; either label can break the exact boundary ties. At $x=0$ , class 1 has twice the density of class 2, so its posterior is $2/3$ . At $x=2$ or $x=-2$ , the class-2 log odds are $0.80685$ , giving a class-2 posterior of about $69.14\%$ .

Fictional equal-prior Gaussian classes share mean zero and have variances 1 and 4. Their densities cross at x approximately minus and plus 1.360. The class-2 posterior is one third at zero and exceeds one half in both tails.

Original analytic illustration. The narrower class dominates near zero; the wider class dominates both tails. Shading marks the class-2 prediction regions, assuming equal error costs.

The result is a classification statement about this supplied model, not evidence that extreme observations predict a profitable trade. Both class densities become small far into the tails even while one posterior approaches one: a high relative probability does not establish that either model fits the observation well.

Why LDA cannot reproduce this example

Expanding the score leaves an $\mathbf{x}$ -quadratic term $-\mathbf{x}^\top\Sigma_k^{-1}\mathbf{x}/2$ . Different $\Sigma_k$ generally prevent that term from cancelling between classes, producing quadratic pairwise boundaries. A boundary can have several components or degenerate to a simpler form; QDA does not promise one curved dividing line.

Linear discriminant analysis, or LDA, instead imposes a single covariance

\Sigma

. The common quadratic term and log determinant cancel, leaving score differences linear in the features. In this example, equal means and equal priors make the LDA scores identical everywhere. No choice of a shared variance recovers the two-tail classification.

Stanford's CS229 notes, §4.1.2, develop the shared-covariance Gaussian model. The owned lesson Generative Classifiers: Gaussian Discriminant Analysis and Naive Bayes teaches that shared-covariance case and explicitly identifies QDA as the extension for different class variances. Its preview is public; the full lesson requires membership.

The price of estimating a separate shape

For $p$ features and $K$ classes, each unrestricted symmetric covariance has $p(p+1)/2$ distinct entries. The Gaussian-model parameter counts are:

Model	Means	Covariances	Priors, if estimated
QDA	$Kp$	$Kp(p+1)/2$	$K-1$
LDA	$Kp$	$p(p+1)/2$	$K-1$

For three classes and ten features, this is 197 parameters for QDA versus 87 for LDA, including learned priors. These are counts of model parameters, not a minimum required sample size or a guarantee about predictive performance.

Ordinary fitting estimates means and covariance matrices from the labelled observations in each class. Maximum-likelihood covariance divides centred sums by the class sample size $n_k$ ; an unbiased sample-covariance convention divides by $n_k-1$ . Keep the convention explicit when reproducing a result.

More observations overall do not solve a thin minority class. A centred class data matrix has rank at most $n_k-1$ , so its unregularised covariance is necessarily singular when $n_k\le p$ . Having $n_k>p$ is only a necessary condition for full rank; collinear features can still defeat inversion, and near-zero eigenvalues can make scores unstable well before exact singularity.

Regularised discriminant analysis (RDA) moderates the separate covariance estimates. Friedman's original paper, §§2–4, combines shrinkage toward a pooled covariance with shrinkage toward a scaled identity matrix, and selects its two controls using estimated prediction error. Pooling can reduce noise but can also erase genuine differences between classes; spherical shrinkage stabilises eigenvalues but depends on the feature scales. Choose transformations and regularisation within training and validation data. There is no universal rule that a quadratic boundary is better because it is more flexible.

Using the probabilities in financial data

Priors describe prevalence; thresholds describe decisions. If class 2 is rare, its prior penalty can outweigh its density advantage. Rebalancing training samples and then blindly using their class fractions changes the implied prior. Specify the population and prediction horizon the probabilities are intended to describe; a correction for changed prevalence alone also assumes the within-class feature distributions remain applicable.

Normalisation is not calibration. The posteriors add to one by construction. Gaussian misspecification, unstable covariance estimates or changes in the data can still make them overconfident. Check probability calibration on observations excluded from fitting, alongside class-specific errors and probability losses. If a separate calibrator is fitted, it also needs predictions made outside the classifier's training sample. scikit-learn's calibration documentation explains why reusing fitted predictions biases this step.

Respect the time at which information exists. A next-session label can train a model only after that session has finished. Fit feature transformations, priors, covariances and tuning choices using information available before each evaluation window. Test on later observations rather than a random mixture of past and future; time-series cross-validation guidance explains the dependence problem. Overlapping label horizons additionally require separating training labels whose outcome windows overlap the test period. QDA's closed-form fit does not protect it from look-ahead bias.

Sources

scikit-learn, Linear and Quadratic Discriminant Analysis, §1.2.2 — Gaussian scores, posterior probabilities and the shared-covariance restriction.
Jerome H. Friedman (1989), Regularized Discriminant Analysis, JASA 84(405), 165–175, §§1–4 — classification loss, QDA/LDA, unstable covariance estimates and the original two-parameter RDA construction.
Andrew Ng, updated by Tengyu Ma, CS229 Lecture Notes, §4.1.2 — shared-covariance Gaussian discriminant modelling and sample estimates.
scikit-learn, Probability calibration, §§1.16.1–1.16.2 — calibration assessment and separation of classifier and calibration fitting.
scikit-learn, Cross-validation, §3.1.2.6 — evaluating dependent observations in time order.

From a feature cloud to a class probability

Same average, different tails

Why LDA cannot reproduce this example

The price of estimating a separate shape

Using the probabilities in financial data

Sources

Related terms

Bayes' theorem

Linear discriminant analysis

Look-ahead bias

Mahalanobis distance

Multivariate normal distribution

Regularised discriminant analysis