Group lasso: Definition & Examples

Group lasso is a regression penalty that can set predefined groups of coefficients to zero together. For a quant modelling a return series with several lags of each economic signal, it offers a way to retain or discard a whole signal family. Ordinary Lasso regression selects individual coefficients instead.

The unit of selection is a block

Partition the predictors into non-overlapping groups $g=1,\ldots,G$ . For centred responses and predictors, one convention is

\widehat\beta=\operatorname*{arg\,min}_{\beta} \left\{\frac{1}{2n}\lVert y-X\beta\rVert_2^2 +\lambda\sum_{g=1}^G w_g\lVert\beta_g\rVert_2\right\}, \qquad \lambda\geq0,\quad w_g>0.

Here

n

is the number of observations,

y

the response vector and

X

the predictor matrix. The vector

\beta_g

contains the coefficients assigned to group

g

; an intercept, if fitted separately, is normally unpenalised. The penalty sums Euclidean norms, not squared norms. Its kink when a whole block reaches zero creates group sparsity. Single-coefficient groups recover the L1 penalty. Yuan and Lin, §2.

Weights matter: $w_g=\sqrt{p_g}$ , where $p_g$ is group size, is a common convention. Unit weights are another choice. Changing weights changes the fit. So does changing the loss normalisation without adjusting $\lambda$ : replacing $1/(2n)$ by $1/2$ requires multiplying $\lambda$ by $n$ to preserve this objective's minimiser.

Two groups, one exact calculation

Consider the following synthetic, dimensionless proximal problem, with unit weights:

\min_\beta\;\frac12\lVert z-\beta\rVert_2^2 +2\bigl(\lVert\beta_1\rVert_2+\lVert\beta_2\rVert_2\bigr), \qquad z_1=(3,4),\quad z_2=(0.5,0.5).

This is the orthogonal-design calculation, rather than a general correlated regression fit. With loss $\frac12\lVert y-X\beta\rVert_2^2$ , it follows when $X^\top X=I$ and $z=X^\top y$ , after dropping a constant. It also appears as a proximal-gradient update, with the step size incorporated into the penalty parameter.

For any block, the exact rule is block soft thresholding, applied separately because the groups do not overlap:

\widehat\beta_g= \begin{cases} 0, & \lVert z_g\rVert_2\leq\lambda w_g,\\ \left(1-\dfrac{\lambda w_g}{\lVert z_g\rVert_2}\right)z_g, & \lVert z_g\rVert_2>\lambda w_g. \end{cases}

The first branch includes $z_g=0$ , even when $\lambda=0$ ; never divide by a zero norm. To see the mechanism, fix the output norm $r$ . The closest vector of norm $r$ points along $z_g$ , leaving the scalar problem $\frac12(\lVert z_g\rVert_2-r)^2+\lambda w_gr$ for $r\geq0$ . Its minimiser is $r=\max(\lVert z_g\rVert_2-\lambda w_g,0)$ .

Block	Input norm	Calculation	Output
Group 1	$5$	$(1-2/5)(3,4)$	$(1.8,2.4)$
Group 2	$\sqrt{0.5}\approx0.707$	Norm is below $2$	$(0,0)$

Group 1's output norm is

3

; both its coordinates shrink by the same factor,

0.6

. With the same quadratic loss and $\lambda=2$ , coordinatewise Lasso instead returns

(1,2)

for its first two coefficients. These are different penalties; the common numerical

\lambda

does not make their regularisation strengths universally equivalent.

Group 1 moves from input (3,4) to (1.8,2.4) along the same ray, while coordinatewise Lasso gives (1,2). Group 1 input norm 5 exceeds threshold 2; Group 2 input norm approximately 0.707 is below 2 and gives the zero vector.

Original synthetic illustration with unit group weights and dimensionless coefficients. The left panel compares directions; the right compares input group norms with the threshold.

What the grouping commits you to

Groups must be supplied before fitting; the method does not discover economic families. A lag block, categorical encoding or spline basis can be a sensible unit, but a nonzero group does not establish that every component is informative. Ordinary group lasso does not encourage sparsity within a surviving group. Sparse-group lasso adds an L1 term for that separate task. Simon et al., §§1–2.

The norm is unchanged by an orthogonal rotation within a group, so a corresponding rotation of the design and coefficients preserves the criterion. Arbitrary rescaling does not. Choose a scaling convention, estimate standardisation from training data only, and preserve it in validation. Standardising columns does not make them orthogonal: the one-shot formula above cannot simply be applied to marginal coefficients from a correlated design.

For the ordinary-penalty foundations and a worked example of training-only scaling, see Ridge, Lasso, and Elastic Net (public preview; full lesson requires membership).

Sources

Yuan and Lin (2006), Model selection and estimation in regression with grouped variables, §§1–2: grouped regression, orthonormal contrasts, weights and block thresholding.
Parikh and Boyd, Proximal Algorithms, §§4.2, 6.5.1 and 6.5.4: proximal-gradient updates, Euclidean-norm shrinkage and non-overlapping sums of norms.
Simon, Friedman, Hastie and Tibshirani, A Sparse-Group Lasso, §§1–2: the distinction between groupwise and within-group sparsity.

The unit of selection is a block

Two groups, one exact calculation

What the grouping commits you to

Sources

Related terms

L1 regularization