L2 regularization: Definition & Examples

L2 regularization adds a penalty proportional to the sum of squared model coefficients to the fitting loss. In least-squares regression this produces ridge regression. A quant can use it to stabilise estimated factor exposures when correlated predictors allow large, offsetting coefficients that fit training returns well but react sharply to small data changes.

The penalty deliberately changes the best fit. It can reduce estimation variance at the cost of bias; it does not guarantee better future returns or lower out-of-sample prediction error.

Which objective?

For observations $i=1,\ldots,n$ , predictors $x_{ij}$ , response $y_i$ , slopes $\beta_j$ and intercept $\beta_0$ , use the convention

J(\beta_0,\boldsymbol\beta) =\frac12\sum_{i=1}^{n}\left(y_i-\beta_0-\sum_{j=1}^{p}x_{ij}\beta_j\right)^2 +\frac{\lambda}{2}\sum_{j=1}^{p}\beta_j^2, \qquad \lambda\ge0.

Here L2 means the squared Euclidean norm,

\lVert\boldsymbol\beta\rVert_2^2

. An unsquared norm would define a different penalty. The first term is half the residual sum of squares, not mean squared error. Multiplying this entire objective by two leaves the minimiser unchanged; changing the scaling of only the data loss changes the effective penalty.

The intercept is excluded. With centred predictors, its fitted value is the sample mean of the response. Penalising it would also pull the response level toward zero, making the fit depend on the choice of origin for $y$ . That can be a deliberate alternative model, but it is not the convention here.

A smaller coefficient can win with a larger residual

Consider a synthetic, dimensionless one-coefficient fit profile with no free intercept:

J(\beta)=\frac12(3-\beta)^2+\frac12\beta^2.

This is a normalised quadratic illustration, not an empirical return forecast. The unpenalised data loss is minimised at $\beta=3$ . Under the penalised objective,

J'(\beta)=(\beta-3)+\beta=0 \quad\Longrightarrow\quad \hat\beta=1.5.

The second derivative is $2>0$ , so this is the unique minimum.

Candidate coefficient	Data loss	L2 penalty	Total $J$
$3$	$0$	$4.5$	$4.5$
$1.5$	$1.125$	$1.125$	$2.25$

The perfect training fit loses because its squared coefficient is expensive. The data-loss and penalty columns are separate quantities; the penalised objective is not itself a prediction-error estimate.

The total objective has minimum beta 1.5 and J 2.25, where data loss and L2 penalty each equal 1.125. Beta 3 has data loss zero and total J 4.5.

Synthetic dimensionless profile with lambda one: total objective, data loss and squared-coefficient penalty. Filled points mark evaluated candidates.

Smooth shrinkage, usually without feature selection

For the scalar profile $\tfrac12(z-\beta)^2+\tfrac\lambda2\beta^2$ , the solution is

\hat\beta=\frac{z}{1+\lambda}.

A finite positive

\lambda

reduces a nonzero

z

continuously toward zero. There is no interval of nonzero

z

values that gets thresholded to exactly zero, unlike L1 regularization. If

z=0

, however, the ridge solution is exactly zero even at finite

\lambda

. Ridge does not generally produce sparse models; saying that it can never have a zero coefficient is too strong.

For centred linear-regression data, penalising all slopes gives

\hat{\boldsymbol\beta} =(X^\top X+\lambda I)^{-1}X^\top\mathbf y, \qquad \lambda>0.

The added diagonal term makes the slope problem strictly convex even when predictor columns are linearly dependent. Shrinkage acts along the eigenvector directions of $X^\top X$ : poorly identified directions receive stronger relative shrinkage. With correlated predictors, individual coefficients in the original coordinates need not all decrease monotonically in magnitude. The scalar division rule applies directly to an orthonormal design, not to every regression coefficient separately.

The tuning parameter has a convention and a scale

Two details matter before comparing fitted models:

Loss normalisation. If another implementation minimises $\text{RSS}/(2n)+\lambda_{\mathrm{mean}}\lVert\boldsymbol\beta\rVert_2^2/2$ , the equivalent parameter in this page's sum-loss convention is $\lambda=n\lambda_{\mathrm{mean}}$ . The same numerical tuning parameter can therefore mean different shrinkage across implementations.
Predictor units. Rescaling a predictor changes the coefficient needed to represent the same prediction, and hence changes its squared penalty. Standardising predictors is a common way to impose comparable penalties. Fit those transformations on training data only; use the fitted transformation on validation and future observations. Domain-specific penalty weights are another deliberate choice.

Choose penalty strength using a validation design that respects the timing of financial observations. A lower penalised training objective alone cannot establish that a forecasting or trading model improved; overfitting remains possible when the tuning process repeatedly searches the same validation data.

For a finance application beyond forecasting, Calibration as an Inverse Problem shows how a squared parameter penalty stabilises an underdetermined price-calibration problem. The lesson has a public preview; full teaching requires membership.

Sources

Hastie, Tibshirani and Friedman, The Elements of Statistical Learning, second edition, corrected 12th printing, §3.4.1, pp. 63–64: ridge objectives, scaling, intercept treatment and the matrix solution.
scikit-learn: Ridge: an explicit implementation convention, residual sum of squares plus alpha times the squared coefficient norm. Its alpha equals this page's $\lambda$ for the same unweighted data and intercept treatment.

Which objective?

A smaller coefficient can win with a larger residual

Smooth shrinkage, usually without feature selection

The tuning parameter has a convention and a scale

Sources

Related terms

L1 regularization

Linear regression

Overfitting

Validation set