Principal components regression: Definition & Examples

Principal components regression (PCR) fits a linear regression using a chosen number of leading principal component scores instead of the original predictors. A quant can compress a panel of correlated signals into fewer regression inputs, reducing exposure to unstable directions. The trade-off is that a direction with little predictor variation can carry the response signal.

PCR has an unsupervised first stage: PCA chooses directions from the predictors without seeing the response. The second stage uses ordinary least squares on the retained scores. Choosing the component count using response-based validation does not make the PCA directions themselves supervised. Stanford's PCR notes express the model as regression in these rotated coordinates.

What is fitted?

Let

X_c

be an

n\times p

training design with each column's training mean subtracted, and let

y_c=y-\bar y\mathbf{1}

. Choose whether to scale the columns before PCA: covariance-based and unit-variance-scaled PCR are different models. Here, assume mean-centred, unscaled predictors and no whitening. Centring alone does not imply scaling; the PCA documentation distinguishes these operations explicitly.

Write the thin singular value decomposition as

X_c=U_rD_rV_r^T,

where $r=\operatorname{rank}(X_c)$ and the positive singular values $d_1\ge\cdots\ge d_r$ are in descending order. The first $k\le r$ columns of $V_r$ form $V_k$ . Their scores and regression fit are

\begin{aligned} Z_k&=X_cV_k=U_kD_k,\\ \hat\gamma_k&=(Z_k^TZ_k)^{-1}Z_k^Ty_c,\\ \hat\beta_k&=V_k\hat\gamma_k,\\ \hat y&=\bar y\mathbf{1}+U_kU_k^Ty_c. \end{aligned}

PCR therefore projects the centred response onto the retained score space. It removes the coefficient contributions of the discarded components entirely. This can help with multicollinearity, but the retained subspace imposes a restriction that can introduce bias. Normal errors are not required merely to compute this least-squares fit.

For a new predictor vector $x_*$ , use the training mean $\bar x$ and the same learned directions:

\hat y_* = \bar y+(x_*-\bar x)^T\hat\beta_k.

If scaling was used, apply the training scales as well. Recomputing PCA on the prediction set changes the coordinates in which the regression was fitted.

The component with most variance can miss everything

Consider four original toy observations in arbitrary units:

Observation	Predictor $x_1$	Predictor $x_2$	Response $y$
A	2	1	1
B	−2	1	1
C	2	−1	−1
D	−2	−1	−1

All three columns have mean zero. Do not rescale this example to unit variance. With covariance divisor

n=4

\frac{X_c^TX_c}{4}=\begin{pmatrix}4&0\\0&1\end{pmatrix}.

PC1 is the first coordinate, up to its arbitrary sign. It captures $4/(4+1)=80\%$ of total predictor variance. Yet its score is orthogonal to $y$ :

x_1^Ty=2-2-2+2=0.

One-component PCR consequently estimates a zero slope and zero intercept. Its four fitted responses are zero, giving training sum of squared errors

\mathrm{SSE}_1=1^2+1^2+(-1)^2+(-1)^2=4.

Retaining both components recovers

\hat y=x_2=y

and

\mathrm{SSE}_2=0

. PC2 alone also fits perfectly, but selecting PC2 alone is not the usual rule of retaining the first

k

components. These are exact training results; four noise-free observations establish no forecast advantage.

Four-observation unscaled toy: PC1 explains 80% of predictor variance but has fitted response zero and training SSE 4. PC2 explains 20% and equals y; retaining both PCs gives SSE 0.

Predictor variance ranks directions in X; regression needs directions that explain y. The second panel fits PC2 alone to reveal the signal that one-component PCR discards.

Autoscaling this particular design would make both predictor variances one and produce tied PCA eigenvalues; the unique PC1 ordering used above would disappear. The unscaled convention is part of the example, not a universal preprocessing recommendation.

The scikit-learn PCR/PLS comparison illustrates the same predictive failure mechanism. Partial least squares regression uses response information when constructing components. That difference makes it an alternative to test, not a guarantee that it will outperform PCR.

How many components, and on whose data?

Explained predictor variance measures reconstruction of $X$ , not prediction of $y$ . A high retained percentage cannot by itself justify the regression's component count.

Fit centring, any scaling, PCA and regression inside each training fold, then transform its validation fold with those fitted quantities. Even PCA can leak information when fitted on the complete dataset; the data-leakage guidance explicitly includes it. Tune

k

using prediction loss. If reporting performance after this tuning, use an untouched test set or an outer validation loop around the inner tuning loop; the nested validation example explains the selection bias otherwise. For financial forecasting, preserve the information available at each date in both loops.

Keeping every nonzero component reproduces the OLS projection onto the training design's column space. Unique OLS slopes require full column rank,

r=p

, which for centred predictors requires

p\le n-1

. If

p\ge n

or columns are dependent, OLS slopes are not unique. Under the unscaled convention above, all-

r

PCR supplies the SVD minimum-norm solution, while the training fitted values remain the same OLS projection. Zero singular values cannot be inverted, and equal training fits do not settle prediction quality.

To work through the PCA geometry, see Principal Component Analysis and the SVD View (public preview; full lesson requires membership).

What is fitted?

The component with most variance can miss everything

How many components, and on whose data?

Sources

Related terms

Ordinary least squares

Partial least squares regression