Skip to content

L1 regularization

How the lasso absolute-value penalty produces exact zeros, why soft thresholding works, and what sparsity does not establish about financial signals.

L1 regularization adds a penalty proportional to the sum of absolute coefficient values to a fitting objective. In a return-prediction or factor-exposure regression, it can set some coefficients exactly to zero, leaving a model that uses fewer predictors. Least-squares regression with this penalty is called the lasso.
The aim is to control overfitting, but a sparse fitted model is not evidence that the surviving factors cause returns or will predict them profitably.

The objective needs its scale

One common convention for linear regression is
min⁡β0,β{12n∑i=1n(yi−β0−∑j=1pxijβj)2+λ∑j=1p∣βj∣},λ≥0.\min_{\beta_0,\boldsymbol{\beta}} \left\{ \frac{1}{2n}\sum_{i=1}^{n} \left(y_i-\beta_0-\sum_{j=1}^{p}x_{ij}\beta_j\right)^2 +\lambda\sum_{j=1}^{p}\lvert\beta_j\rvert \right\},\qquad \lambda\geq0.
Here yiy_i is the response, xijx_{ij} is predictor jj at observation ii, and λ\lambda controls the penalty. The intercept β0\beta_0 is unpenalised in this convention. The sum of absolute values is the L1 norm, not a count of nonzero coefficients.
Increasing λ\lambda puts more weight on reducing coefficient magnitudes relative to data fit. At λ=0\lambda=0 the objective reduces to ordinary least squares. A numerical penalty setting only makes sense alongside its objective: replacing the averaged loss above by the residual sum of squares requires multiplying the penalty coefficient by 2n2n to keep the same minimisers. Software parameter names alone do not establish equivalent settings.

Why a coefficient can become exactly zero

Consider the synthetic, dimensionless scalar problem

J(β)=12(z−β)2+λ∣β∣.J(\beta)=\frac12(z-\beta)^2+\lambda\lvert\beta\rvert.
The absolute-value term has a kink at zero. For β>0\beta>0, the derivative of JJ is β−z+λ\beta-z+\lambda; for β<0\beta<0, it is β−z−λ\beta-z-\lambda. At zero, the absolute value has subgradients filling the interval [−1,1][-1,1]: each is a slope of a supporting line. Consequently,
0∈−z+λ[−1,1]⟺∣z∣≤λ.0\in -z+\lambda[-1,1] \quad\Longleftrightarrow\quad \lvert z\rvert\leq\lambda.
This is the optimality condition for choosing zero. Combining it with the two nonzero branches gives soft thresholding:
β^={z−λ,z>λ,0,∣z∣≤λ,z+λ,z<−λ.\hat\beta= \begin{cases} z-\lambda,&z>\lambda,\\ 0,&\lvert z\rvert\leq\lambda,\\ z+\lambda,&z<-\lambda. \end{cases}

Large positive or negative responses move towards zero by λ\lambda; responses within the threshold become exactly zero. The quadratic term makes this scalar objective strictly convex, so the minimiser is unique, including at either threshold.

With z=3z=3 and λ=1\lambda=1, the optimum is β^=2\hat\beta=2:

Candidate coefficientData loss 12(3−β)2\tfrac12(3-\beta)^2Penalty ∣β∣\lvert\beta\rvertFull objective J(β)J(\beta)
004.54.5004.54.5
220.50.5222.52.5
33003333
The unpenalised solution is 33, with zero data loss. Its value of 3 in the last column comes from evaluating it in the penalised objective. Lasso accepts a worse data fit to reduce the combined criterion. Change only the response to z=0.5z=0.5 and the same penalty chooses β^=0\hat\beta=0, with total objective 0.1250.125.
For lambda one, response three gives coefficient two and full objective 2.5; evaluating coefficient three gives objective three. The soft-threshold curve maps every response between minus one and one to zero, including 0.5.
For lambda one, response three gives coefficient two and full objective 2.5; evaluating coefficient three gives objective three. The soft-threshold curve maps every response between minus one and one to zero, including 0.5.
Original dimensionless scalar illustration. Filled markers are penalised minimisers; the hollow marker is the unpenalised solution evaluated in the full objective. The dashed line shows the unpenalised coefficient as the response changes.

This closed form applies to the scalar problem and, coordinate by coordinate, to appropriately normalised orthogonal least-squares designs. With correlated predictors, the coefficients interact: thresholding each ordinary least-squares coefficient independently does not generally solve the lasso problem.

What to inspect in a financial model

Predictor units affect the penalty. A change from returns expressed as fractions to returns expressed as percentages changes the compensating coefficient and its absolute-value penalty. Choose an explicit scaling convention, estimate any centring and scaling from training observations alone, and apply it unchanged to validation observations.

Multicollinearity makes selection harder to interpret: strongly related signals can substitute for one another, and small data changes can alter which survives. General lasso coefficient solutions need not be unique. A zero is a decision of this fitted objective, not proof that a variable is economically irrelevant.

Choose the penalty using validation appropriate to the prediction horizon and information timing. Regularisation cannot repair information leakage, and a lower penalised training objective is not a guarantee of better out-of-sample error or returns after costs. L2 regularization provides a useful comparison: its smooth quadratic penalty generally shrinks coefficients without creating L1's interval of exact zeros.
For a worked comparison of the penalties and a training-only scaling example, see Ridge, Lasso, and Elastic Net (membership; public preview).

Sources

Last updated
  • machine learning
  • lasso
  • regularisation
  • sparsity
  • factor models
Keep connecting the ideas
← Back to the glossaryBack to top ↑