Partial least squares regression: Definition & Examples

Partial least squares regression (PLS) predicts a response using a small set of linear combinations of the predictors, with the response helping choose those combinations. A quant forecasting a future return from many related signals can use PLS to compress the signal panel toward directions associated with that target. The association must be learned from past training data; it is not evidence that the signals cause returns.

The distinguishing feature is supervised compression. Principal components regression first compresses predictors by their own variance, then regresses the response on the retained components. PLS lets the response influence the compression itself. Keeping fewer components can make a large, correlated predictor set manageable, but also creates a model choice that needs validation. The scikit-learn comparison illustrates why a low-variance predictor direction can matter for a target.

The first component listens to the response

Consider PLS1, meaning one response variable. Let

X

be an

n\times p

predictor matrix and

y

n

-vector of responses, both mean-centred using training-sample means. Fix a predictor scaling convention before fitting. For the first component of the regression-deflation convention used here, choose a unit-length weight vector to maximise squared sample covariance with the response:

\max_{\lVert w\rVert_2=1}\left(w^\top X^\top y\right)^2.

The covariance divisor is constant, so it does not affect the maximiser. If $X^\top y\ne0$ , one choice of sign gives

w_1=\frac{X^\top y}{\lVert X^\top y\rVert_2}, \qquad t_1=Xw_1, \qquad q_1=\frac{t_1^\top y}{t_1^\top t_1}.

Here

w_1

contains weights,

t_1

is the component's score across observations, and

q_1t_1

is its fitted response contribution. Reversing the sign of

w_1

t_1

and

q_1

leaves that contribution unchanged. This objective fixes the length of the weights, not the variance of the score: maximising covariance is not the same as maximising correlation.

A quiet predictor can contain the whole target

Take four fictional observations, with arbitrary numerical units:

Observation	Predictor $x_1$	Predictor $x_2$	Response $y$
A	2	1	1
B	−2	1	1
C	2	−1	−1
D	−2	−1	−1

These columns already have mean zero. Do not standardise them for this example. Using variance divisor

n=4

\frac{X^\top X}{4} =\begin{pmatrix}4&0\\0&1\end{pmatrix}, \qquad X^\top y=\begin{pmatrix}0\\4\end{pmatrix}.

PCA's first direction is the

x_1

coordinate: it captures

4/(4+1)=80\%

of predictor variance. Yet

x_1^\top y=0

. An ordinary least squares regression on that one score, with an intercept, predicts zero for every observation and has training sum of squared errors

1+1+1+1=4

PLS1 instead chooses $w_1=(0,1)^\top$ . Its score is $t_1=x_2=y$ , its response coefficient is $q_1=1$ , and its one-component fitted values reproduce all four responses. Training squared error is zero.

Four fictional centred, unscaled observations. PCA1 follows x1 with variance 4 and captures 80 percent of predictor variance, but one-component PCR has training SSE 4. PLS1 follows x2 with variance 1, equal to y, and has training SSE 0.

Figure: The direction with the largest predictor variance need not be the direction associated with the response. The example compares exact training fits under a shared, unscaled preprocessing convention.

The result is deliberately constructed, not a forecast experiment or a general ranking of PLS over PCR. PCR with both components also recovers

y

. Moreover, scaling both predictor columns to unit variance makes their covariance matrix the identity: there is then no unique leading PCA direction. The unscaled comparison cannot be carried over unchanged. In software, check this explicitly: scikit-learn's PLSRegression centres inputs and defaults to scaling both predictors and responses; its scale=False setting preserves this example's convention.

Further components require a deflation rule

For the PLS1 regression-deflation convention, remove each fitted score's contribution from both predictors and response. Starting with

E=X

and

f=y

, a step with score

t=Ew

uses

p=\frac{E^\top t}{t^\top t}, \qquad q=\frac{t^\top f}{t^\top t}, \qquad E\leftarrow E-tp^\top, \qquad f\leftarrow f-tq.

The next weight is proportional to $E^\top f$ , if nonzero. Stop when the chosen component count is reached or no usable response covariance remains. In the toy example $f=0$ after the first step, so extraction stops.

Later weights act on deflated predictors. They are not directly the coefficients on the original columns; prediction must preserve the fitted sequence of transformations and loadings. Restore the training response mean, and undo any response scaling, to report predictions in the original units. With multiple responses the method is commonly called PLS2. Response scaling and the algorithm matter: PLS regression, canonical PLS and SVD-based cross-decomposition have different extraction or deflation rules. State the variant rather than treating every result labelled PLS as interchangeable. The official cross-decomposition guide sets out these distinctions.

Put the entire extraction inside validation

PLS looks at $y$ while choosing its components. Extracting components once from the full dataset and then cross-validating only the final regression exposes validation responses to the model. That is preprocessing leakage, even when the final regression never sees those responses directly.

For each training fold, learn imputation, centring, any scaling, the response-guided components and the response regression afresh. Transform the validation predictors using those fitted training transformations; do not use validation responses to construct their scores. Tune the component count through cross-validation within the training sample, leaving an outer test period untouched for final assessment. For a future-return forecast, preserve chronology and respect when the response becomes observable, including overlapping return horizons. The scikit-learn leakage guidance explains the train-only fitting principle.

A component can capture a chance target association just as a raw predictor can. Few components do not eliminate overfitting, and a large component weight is not a causal effect or an ordinary regression significance test.

For the variance-compression side of the comparison, SelfMadeQuant's PCA and Factor Compression derives PCA from the covariance eigenbasis and distinguishes panel reconstruction from forecasting. The public page previews the lesson; full teaching requires membership.

Sources

Scikit-learn: Cross decomposition, especially PLSCanonical and PLSRegression — component weights, scores, loadings and deflation variants.
Scikit-learn: PLSRegression API — PLS1/PLS2 terminology, centring and scaling controls.
Scikit-learn: Principal Component Regression vs Partial Least Squares Regression — supervised versus unsupervised compression; its simulated example differs from the exact four-row example above.
Scikit-learn: Common pitfalls, data leakage — fit transformations only on training subsets.
Scikit-learn: Cross-validation of time series data — chronology-aware evaluation and separating model selection from final testing.

The first component listens to the response

A quiet predictor can contain the whole target

Further components require a deflation rule

Put the entire extraction inside validation

Sources

Related terms

Covariance

Ordinary least squares

Overfitting

Principal components regression