Dimensionality reduction: Definition & Examples

Dimensionality reduction represents observations using fewer coordinates than their original feature vectors. A quant researcher might replace a large panel of correlated market measurements with a smaller set of summaries, making a risk model or forecasting pipeline easier to estimate and inspect. The important question is what the smaller representation preserves: fewer columns alone do not establish that useful information survived.

Write the transformation as

f:\mathbb{R}^{p}\longrightarrow\mathbb{R}^{k},\qquad k<p.

Each observation starts with $p$ features and ends with $k$ coordinates. The number of observations need not change. Sampling fewer trading dates is a different operation.

A case where nothing is lost

Suppose two synthetic, dimensionless features are exact duplicates: every observed pair has the form $(t,t)$ . Keeping the first coordinate gives $t$ , and the inverse rule $t\mapsto(t,t)$ recovers the entire pair. Two recorded features describe a set with only one freely varying coordinate.

This is a simple feature-selection example: retain an original feature. Feature extraction instead constructs new coordinates from the original ones. For example, choose the unit direction

\mathbf{u}=\frac{1}{\sqrt{2}}(1,1)

and use the signed coordinate along that direction:

s=\mathbf{u}^{\mathsf T}\mathbf{x} =\frac{x_1+x_2}{\sqrt{2}}.

For $\mathbf{x}=(2,2)$ , the new coordinate is $s=2\sqrt{2}$ , and reconstructing gives $s\mathbf{u}=(2,2)$ . Every point on the duplicate line can be recovered this way. The exact recovery claim applies to that restricted set, not to arbitrary observations in the plane.

Exact duplicate dimensionless features lie on x2=x1. Retaining x1 recovers those pairs. Off the line, (3,1) and (1,3) have the same score 2 square root of 2 and both reconstruct to (2,2).

Original synthetic geometry with a fixed origin and equally scaled, dimensionless axes. Filled markers are observations; the hollow marker is their common reconstructed projection. Exact recovery on the duplicate line does not extend to the whole plane.

The right panel exposes the limitation. Both

(3,1)

and

(1,3)

produce

s=2\sqrt{2}

. Their orthogonal projections are the same point,

(2,2)

. The score cannot tell which original observation generated it: the difference between the two feature values has been discarded.

What PCA chooses to preserve

Principal component analysis (PCA) learns orthogonal directions from the data. For centred features, retaining the leading directions maximises retained sample variance among linear orthogonal projections of the chosen dimension. Equivalently, it minimises total squared Euclidean reconstruction error in the coordinates being analysed.

The example above is a fixed geometric projection. In centred PCA, the score is calculated from $\mathbf{x}-\boldsymbol{\mu}_{\mathrm{train}}$ , where $\boldsymbol{\mu}_{\mathrm{train}}$ is the fitted training mean. Its $2\sqrt{2}$ score therefore also describes a PCA score only when the origin, direction and scaling match—for example, when that direction is retained and the training mean is zero. A different fitted mean changes the score of the raw point $(2,2)$ .

Scaling also changes what “large variance” means. A PCA of raw features measured in different units can emphasise a feature because of its numerical scale. Standardising features is one possible choice, not an automatic requirement or a guarantee of economic relevance. Decide whether absolute variation or variation relative to each feature’s scale suits the task.

The Principal Component Analysis and the SVD View lesson develops the maximum-variance direction and projection construction. Its public page is a preview; the full lesson requires membership.

Compression, prediction and pictures answer different questions

For a risk summary, a researcher may want to reconstruct movements across a yield curve or asset panel with few factors. For a forecast, the objective is performance on a future target. PCA does not use that target to choose its directions. A low-variance feature can carry useful predictive information, so a high explained-variance share does not certify forecasting quality. Reducing dimension can help limit overfitting, but it can also remove the signal the model needs.

A visualisation has another purpose. Nonlinear methods such as t-SNE aim to reveal aspects of neighbourhood structure in a small number of plotting coordinates; they do not share PCA’s linear reconstruction objective. Apparent gaps between plotted groups are not automatically faithful distances in the original feature space. The Nonlinear Dimensionality Reduction lesson explains LLE and t-SNE, including their use as exploratory tools. A public preview is available; the full lesson requires membership.

Keep the transformation inside the experiment

When evaluating a predictive pipeline, split the data before learning means, scales, retained directions or selected features. Fit those transformations on the training set, then apply the fitted rules to validation and test data. Within cross-validation, refit them inside each training fold. Choosing the retained dimension is also a model-selection decision; keep the final test set out of that choice.

For financial time series, the split and each training window must respect information availability. Fitting PCA to the entire history before a historical backtest lets later observations influence earlier representations, even though PCA uses no forecast labels. Unsupervised fitting can still introduce look-ahead bias.

Sources

James, Witten, Hastie and Tibshirani, An Introduction to Statistical Learning with Applications in R, second edition (2021), §§6.3.1 and 12.2.1–12.2.4: centred component scores, variance and reconstruction interpretations, scaling and the limits of principal components regression. Authors’ book page.
Scikit-learn: PCA, for the fitted linear transformation and the distinction between centring and scaling.
Scikit-learn: common pitfalls, for fitting preprocessing only on training data, including PCA.
Scikit-learn: t-SNE, for its visualisation purpose and limits on preserving global structure.

A case where nothing is lost

What PCA chooses to preserve

Compression, prediction and pictures answer different questions

Keep the transformation inside the experiment

Sources

Related terms

Feature selection

Look-ahead bias

Orthogonal projection

Overfitting

Test set

Training set