How shared Gaussian covariance produces linear class boundaries, why priors move them, and how LDA classification differs from Fisher projection and logistic regression.
Linear discriminant analysis
Linear discriminant analysis (LDA) is a generative classifier that models each labelled class with a Gaussian feature distribution, gives all classes a common covariance matrix, and uses Bayes' rule to calculate class probabilities. The shared covariance makes its decision boundaries linear. For a quant comparing labelled liquidity or market conditions, LDA supplies a transparent baseline: separation comes from differences in class means, measured against a common pattern of feature variation.
The name also refers to a closely related supervised projection method. A projection, a class probability and a trading decision are different outputs; understanding their connection prevents several common misuses.
Why a Gaussian model gives a straight boundary
For classes k=1,…,K, LDA assumes
X∣Y=k∼N(μk,Σ),P(Y=k)=πk>0,k∑πk=1.
The multivariate normal distribution describes features within each class; it need not describe the pooled observations. The common positive definite covariance matrix Σ permits correlated features. LDA does not assume feature independence or equal variances across different features.
Bayes' theorem combines each class density with its prior. Expanding the Gaussian exponent produces a quadratic term −x⊤Σ−1x/2 and a log-determinant term shared by every class. Both cancel when comparing classes. The remaining score is
δk(x)=x⊤Σ−1μk−21μk⊤Σ−1μk+logπk.
Each score is affine: a linear feature combination plus an intercept. Setting δk(x)=δj(x) gives a hyperplane whenever the two score slopes differ. The classifier selects the largest score, with a stated rule for ties. This is the shared-covariance construction developed in Stanford's CS229 notes, §§4.1.2–4.1.3 and the official discriminant-analysis derivation.
Equivalently, compare distances to the class means using the common Mahalanobis distance, adjusted for priors. Nearest mean in ordinary Euclidean distance is generally a different rule.
Linear log odds, curved probability
Consider a fictional population model, not an estimated market dataset. A dimensionless feature has distributions
Under equal error costs, the boundary is x=1. At x=1.5, the log odds are 1, the odds are e:1, and the class-2 posterior is 0.731059, or approximately 73.11%. Linear describes the log odds and boundary, not the probability curve.
Now change only the priors to π2=0.1 and π1=0.9. The log odds become 2x−2−log9. The boundary moves to 1+21log9≈2.0986, and the class-2 posterior at x=1.5 falls to approximately 23.20%. A feature closer to class 2's mean can still receive class 1's label because class 2 is rarer.
Fictional Gaussian classes have means 0 and 2 and common variance 1. With equal priors the class-2 probability crosses one half at x equals 1 and reaches 0.731 at x equals 1.5. With class-2 prior 0.1 the crossing moves to 2.099 and the probability at 1.5 is 0.232.
Original analytic illustration. Left: the supplied class densities. Right: model posteriors for two prior choices, holding those densities fixed. Boundaries use equal error costs.
The equal-prior posterior has the same sigmoid form as binary logistic regression. The fitting principles differ: LDA estimates class distributions and priors; logistic regression fits the conditional class probability directly. Their fitted coefficients need not agree on a finite dataset.
Fisher's direction is a projection, not a probability
For two classes with distinct means, Fisher's linear discriminant chooses a nonzero direction w that maximises squared separation of projected means relative to within-class variation. With a common positive definite within-class covariance SW, the criterion is
J(w)=w⊤SWw[w⊤(μ2−μ1)]2,w∝SW−1(μ2−μ1).
Multiplying w by a nonzero constant leaves the criterion unchanged. A within-class scatter matrix proportional to SW gives the same direction. The separation-to-variation objective comes from Fisher's original paper, §§I–II.
This moment-based optimisation does not itself require Gaussian densities. Under the shared Gaussian model, its direction matches the normal to the binary LDA boundary. The projected value z=w⊤x is still just a score: turning it into a label requires a threshold; turning it into a posterior requires additional distributional and prior assumptions.
For K classes, supervised LDA dimension reduction retains at most min(p,K−1) discriminant directions, where p is the feature count. It uses class labels to seek between-class separation relative to within-class variation. Principal component analysis instead seeks directions of large total variance without using labels. A high-variance direction can therefore be irrelevant for class separation. The official dimension-reduction discussion explains the K−1 limit.
Also distinguish latent Dirichlet allocation, the unrelated topic model that shares the acronym LDA. Blei, Ng and Jordan's original paper models collections such as documents through latent topics.
What must be learned before the forecast
Given labelled training observations, estimate each class mean and pool residuals around their own class means. The joint maximum-likelihood covariance convention is
ΣML=n1i=1∑n(xi−μyi)(xi−μyi)⊤.
An unbiased pooled estimate instead uses divisor n−K when all K classes are represented. Neither is the covariance around the overall mean, which also contains between-class variation. State the estimator convention when reproducing probabilities.
Pooling reduces covariance parameters relative to quadratic discriminant analysis, which fits a separate covariance per class. It does not make covariance estimation disappear: the pooled residual matrix has rank at most n−K. An ordinary inverse is impossible when that rank is below p, and collinearity can cause singularity even with more observations. Regularised discriminant analysis explains ways to stabilise covariance estimates; the appropriate degree of pooling remains an empirical question.
For market observations, fit imputation, transformations, feature selection, class parameters and any learned priors inside each training window, then apply the fitted choices to later evaluation observations. Supervised projections also use labels and must be fitted inside that window. Randomly mixing past and future can create look-ahead bias; a label covering future returns is unavailable until its outcome window ends. Separate overlapping training and test label windows as well as ordering feature timestamps. The official guides discuss training-only preprocessing and time-series evaluation.
Choose priors for the intended prediction population. Oversampling a rare condition does not make it common in deployment; changing priors alone is appropriate only if the class-conditional feature distributions remain applicable. Separately, maximum-posterior classification assumes equal misclassification costs. With unequal costs, choose the action with the lowest posterior expected loss.
Finally, probabilities that sum to one need not be empirically reliable. Assess probability calibration on held-out predictions, especially when class shapes, tails or prevalence change. Any fitted recalibration also needs observations excluded from the base classifier's fit, as the official calibration guidance explains. A well-separated class or a high posterior alone does not establish an economic benefit after costs.
