Reduced-rank regression: Definition & Examples

Reduced-rank regression fits a multivariate linear model while restricting the rank of its coefficient matrix. Several response forecasts must then share a small number of linear predictor scores. For a quant modelling many sector returns from a common predictor set, this is a way to impose a common forecasting structure instead of estimating every response equation independently.

The restriction concerns the predictable component. It does not assert that observed returns, or their residuals, have low rank.

One rank constraint joins the response equations

Let

X

be an

n\times p

predictor matrix and

Y

n\times q

response matrix. Here both are column-centred; the intercept is fitted separately and is excluded from the rank restriction. The original-scale intercept vector is

\widehat\alpha=\bar y-\widehat B^\top\bar x

, with means written as column vectors.

For a chosen integer $0\le r\le\min(p,q)$ , the unweighted least-squares problem is

\widehat B_r\in\underset{\operatorname{rank}(B)\le r}{\operatorname{argmin}} \lVert Y-XB\rVert_F^2, \qquad \lVert Y-XB\rVert_F^2 =\sum_{i=1}^n\sum_{j=1}^q\bigl(Y_{ij}-(XB)_{ij}\bigr)^2.

The squared Frobenius norm sums squared errors across observations and responses. Changing one response from percentage points to basis points changes its weight in this loss unless the scaling is adjusted deliberately.

Without the constraint, multivariate linear regression using a common design gives the same coefficients as separate OLS fits. The rank constraint couples those fits: all columns of

B

must lie in a shared subspace of dimension at most

r

Equivalently, write $B=AC^\top$ , where $A$ is $p\times r$ and $C$ is $q\times r$ . The scores $XA$ feed all response forecasts through $C$ . These factors are not uniquely labelled economic forces: for any invertible $r\times r$ matrix $H$ , replacing $A$ with $AH$ and $C$ with $CH^{-\top}$ leaves $B$ unchanged.

Truncate the fitted responses

Assume first that centred $X$ has full column rank. Define

\widehat B_{\mathrm{OLS}}=(X^\top X)^{-1}X^\top Y, \qquad \widehat Y=X\widehat B_{\mathrm{OLS}}=P_XY.

Take the singular value decomposition of $\widehat Y$ , and let $V_r$ contain its first $r$ right singular vectors. A solution is

\widehat B_r=\widehat B_{\mathrm{OLS}}V_rV_r^\top, \qquad X\widehat B_r=\widehat YV_rV_r^\top.

Why this matrix? Orthogonality of OLS residuals gives

\lVert Y-XB\rVert_F^2 =\lVert Y-\widehat Y\rVert_F^2 +\lVert\widehat Y-XB\rVert_F^2.

The first term is fixed. The second is minimised by the best rank-

r

approximation to the fitted matrix. If its singular values are

d_1\ge d_2\ge\cdots

, the minimum loss is the full OLS loss plus

\sum_{k>r}d_k^2

. Bunea, She and Wegkamp give this construction in §2.1.

For rank-deficient $X$ , a pseudoinverse gives a representative coefficient solution; fitted values can be identified while coefficients are not unique. Ties at the singular-value cutoff can also make the retained subspace nonunique.

Four rows: what rank one costs

Use the exact fictional centred matrices

X=\begin{pmatrix} 0.5&0.5\\-0.5&0.5\\0.5&-0.5\\-0.5&-0.5 \end{pmatrix}, \qquad Y=X\begin{pmatrix}3&0\\0&1\end{pmatrix}.

Here $X^\top X=I$ . The full OLS coefficient matrix is $\operatorname{diag}(3,1)$ and fits $Y$ exactly. Its fitted response matrix has singular values $3$ and $1$ . Rank one keeps the larger direction:

\widehat B_1=\begin{pmatrix}3&0\\0&0\end{pmatrix}.

The first response stays exact. The second is predicted as zero, leaving residuals $0.5,0.5,-0.5,-0.5$ . Thus the total squared error rises from $0$ to $4(0.5)^2=1$ . No other rank-one coefficient matrix achieves a smaller loss under this design and loss.

Exact fictional four-row example. Observed response pairs are (1.5, 0.5), (−1.5, 0.5), (1.5, −0.5) and (−1.5, −0.5). Rank one sets response 2 to zero, leaving four residuals of magnitude 0.5 and SSE 1. The full fit has SSE 0; fitted singular values 3 and 1 contribute squared values 9 and 1.

Exact fictional data with an orthonormal design. Rank one keeps the first fitted response direction and discards the second; the discarded squared singular value is the increase in training SSE.

This tiny construction checks the optimisation algebra. It provides no evidence about forecast accuracy or the number of factors in market data.

The coefficient-SVD shortcut can fail

The example makes truncating $\widehat B_{\mathrm{OLS}}$ look sufficient because $X^\top X=I$ . Generally the relevant error is $\lVert X(B-\widehat B_{\mathrm{OLS}})\rVert_F^2$ , so predictor geometry matters. Use the fitted-response construction above, or an equivalent correctly whitened problem.

For a concrete counterexample, replace the first column of the toy $X$ by one tenth of itself, obtaining $\widetilde X$ , and set $\widetilde Y=\widetilde X\operatorname{diag}(3,1)$ . OLS coefficients remain $\operatorname{diag}(3,1)$ , but fitted singular values are now $1$ and $0.3$ . The optimal rank-one coefficients are $\operatorname{diag}(0,1)$ with loss $0.3^2=0.09$ . Naively retaining the largest coefficient singular value gives $\operatorname{diag}(3,0)$ and loss $1$ . Even orthogonal predictor columns are insufficient for that shortcut when their lengths differ.

Rank is a modelling choice

A smaller rank sacrifices training fit to restrict the forecasting structure; improvement on future observations is a separate question. Select rank and preprocessing inside training data, then assess forecasts on held-out data. Financial time series need validation that respects dates and overlapping outcomes. A low-rank fit alone does not resolve multicollinearity, establish causality or justify ordinary OLS standard errors. The fitted subspace is selected using

Y

, so counting free matrix entries is not automatically the estimator's effective degrees of freedom; Mukherjee and colleagues examine this distinction.

Principal components regression chooses predictor directions using variation in

X

; reduced-rank regression chooses a shared forecasting structure using

X

and

Y

. Canonical correlation analysis instead maximises correlations between scores from two variable blocks. Related matrix techniques can therefore answer different questions.

For the underlying geometry, the Singular Value Decomposition lesson explains singular vectors, rank and squared matrix size. It has a public preview; the full lesson requires membership.

Sources

Florentina Bunea, Yiyuan She and Marten H. Wegkamp (2011), Optimal selection of reduced rank estimators of high-dimensional matrices, §2.1 — projected-response construction, coefficient recovery and discarded-singular-value loss.
Ashin Mukherjee and Ji Zhu (2011), Reduced Rank Ridge Regression and Its Kernel Extensions, §§1–2 — latent-factor motivation, collinearity limitations and the distinction between rank reduction and ridge regularisation.
Ashin Mukherjee, Kun Chen, Naisyin Wang and Ji Zhu (2015), On the degrees of freedom of reduced-rank estimators in multivariate regression, §§1 and 3 — centred multivariate model, rank constraint, fitted-response SVD and effective-complexity caveats.

The four-row illustration, rescaled-design counterexample and figure are original deterministic examples.

One rank constraint joins the response equations

Truncate the fitted responses

Four rows: what rank one costs

The coefficient-SVD shortcut can fail

Rank is a modelling choice

Sources

Related terms

Canonical correlation analysis

Multivariate linear regression

Principal components regression