Skip to content

Stability selection

Stability selection retains frequently selected variables across subsamples. Distinguish empirical frequencies, half-sampling and assumption-dependent PFER bounds.

Stability selection repeatedly applies a feature-selection procedure to subsamples and retains variables selected sufficiently often. For a quant comparing candidate return predictors, it exposes a difference between a variable that survives many perturbations of the training data and one that appears in a single fitted model.
It wraps a base feature-selection method, such as the lasso. The selected names remain conditional on that method, its tuning settings and the available history.

Six subsamples, one retained feature

Consider six stipulated subsamples at one fixed selector setting. Suppose the resulting sets are A and B on each of the first four runs, A and C on the fifth, and no features on the sixth. These are constructed outcomes, not fitted market data.
CandidateTimes selectedEmpirical frequencyRetained at a threshold of 0.75?
A5 of 65/6≈0.83335/6\approx0.8333Yes
B4 of 64/6≈0.66674/6\approx0.6667No
C1 of 61/6≈0.16671/6\approx0.1667No
Use the convention retain when frequency is at least the threshold. Since 6×0.75=4.56\times0.75=4.5, an integer count must be at least five. Only A qualifies. A rule using a strict inequality differs when a frequency lands exactly on the threshold, so the boundary convention belongs in the procedure.
At one fixed selector setting on six stipulated subsamples, A is selected five times, B four times and C once. Their frequencies are 5/6, 4/6 and 1/6. Only A clears the 0.75 threshold.
At one fixed selector setting on six stipulated subsamples, A is selected five times, B four times and C once. Their frequencies are 5/6, 4/6 and 1/6. Only A clears the 0.75 threshold.
Original stipulated example. A filled cell means selected on that subsample. The dashed line is the inclusive 0.75 frequency threshold; this table alone supplies no false-selection guarantee.
A's frequency is not an 83.33% probability that A is economically real, causal or profitable. The table records what a particular algorithm did on particular subsamples. Neither the response values nor any ground-truth signal labels were stipulated, so it cannot identify a false discovery.

What probability does the frequency estimate?

In the classical construction, draw II uniformly without replacement with size ⌊n/2⌋\lfloor n/2\rfloor. Given observed data DD, let S^λ(I)\widehat S^\lambda(I) be the base-selected set and define

πjλ(D)=P∗ ⁣(j∈S^λ(I)∣D),π^j,Bλ=1B∑b=1B1{j∈S^λ(Ib)}.\pi_j^\lambda(D) =\mathbb{P}^{*}\!\left(j\in\widehat S^\lambda(I)\mid D\right), \qquad \widehat\pi_{j,B}^\lambda =\frac1B\sum_{b=1}^{B} \mathbf{1}_{\{j\in\widehat S^\lambda(I_b)\}}.
Here BB is the number of subsamples and jj indexes a candidate variable. The first quantity is the exact resampling probability; the second is a finite-run estimate. The star includes algorithm randomisation if present. Over a specified tuning set Λ\Lambda, the rule can threshold max⁡λ∈Λπjλ(D)\max_{\lambda\in\Lambda}\pi_j^\lambda(D), estimated by its frequency counterpart. Meinshausen and Bühlmann, §§2.1–2.3, give this construction.

Half-subsampling uses distinct observations within each draw. An ordinary bootstrap samples with replacement, typically drawing nn rows, so observations can repeat. A bootstrap frequency need not inherit a half-subsampling theorem.

A separate bound on false selections

The classical bound concerns the expected number of noise variables retained, E[V]\mathbb{E}[V], called the per-family error rate (PFER). Under the paper's i.i.d. half-subsampling setup, exchangeable noise-selection indicators and its “not worse than random guessing” condition on the base procedure, Theorem 1 gives
E[V]≤qΛ2(2πthr−1)p,12<πthr<1,qΛ=E ⁣[∣⋃λ∈ΛS^λ(I)∣].\mathbb{E}[V] \leq\frac{q_\Lambda^2}{(2\pi_{\mathrm{thr}}-1)p}, \qquad \frac12<\pi_{\mathrm{thr}}<1, \qquad q_\Lambda=\mathbb{E}\!\left[ \left\lvert\bigcup_{\lambda\in\Lambda}\widehat S^\lambda(I)\right\rvert \right].

Here pp counts all candidate variables; qΛq_\Lambda is the expected base-selection union size on a half-sample, not the final stable-set size. With one setting it reduces to the expected base-selected count. Noise exchangeability concerns the joint selection behaviour of noise variables under permutations; ordinary predictor correlation does not establish it.

For a separate hypothetical procedure satisfying those conditions, suppose p=100p=100, qΛ=5q_\Lambda=5 and πthr=0.9\pi_{\mathrm{thr}}=0.9. Then
E[V]≤52(2×0.9−1)100=2580=0.3125.\mathbb{E}[V]\leq \frac{5^2}{(2\times0.9-1)100} =\frac{25}{80}=0.3125.

This is an upper bound on an expectation across repeated data sets and procedure randomness. It is not a realised fractional variable count, a 31.25% false-discovery fraction, or a guarantee about A. The six-run example neither supplies qΛ=5q_\Lambda=5 nor establishes the assumptions.

The original bound uses ideal resampling probabilities. Arbitrary finite frequencies do not automatically receive that guarantee. Complementary pairs stability selection (CPSS) uses pairs of disjoint half-samples. Shah and Samworth, §§1–3.1, derive finite-pair bounds on low-selection-probability variables without noise-exchangeability assumptions; identifying those variables with true noise needs additional conditions. Their result is not a blanket guarantee for any resampling design.
False discovery rate instead concerns an expected false-selection proportion. A PFER calculation should be reported with its own target and assumptions.

What stability means in a financial workflow

Correlated signals can substitute for each other: a base selector may alternate between related predictors, leaving neither above the threshold. A low individual frequency therefore need not mean the underlying economic exposure is useless. Ridge, Lasso, and Elastic Net develops this base-selector behaviour (membership; public preview).
Keep selection, learned preprocessing and any tuning within the eligible training set. Assess the complete procedure on later observations, as developed in Time-Ordered Train, Validation and Test Design (membership; public preview).
Financial dependence also matters to the theorem: random row halves can break serial structure or separate overlapping return labels. Choosing blocks may be appropriate to the research design, but changes the procedure and requires its own justification. A stable selection list alone certifies neither the i.i.d. setup nor performance after transaction costs.

Sources

  • Meinshausen, N. and Bühlmann, P. (2010), Stability selection, §§2.1–2.4, Definitions 1–3 and Theorem 1: half-subsampling, stable sets, qΛq_\Lambda and the assumption-dependent PFER bound. The author preprint also exposes the equations as searchable text.
  • Shah, R. D. and Samworth, R. J. (2013), Variable selection with error control: another look at stability selection, §§1–3.1, Definition 1 and Theorem 1: population versus finite-resampling control, CPSS and the distinction between noise and low base-selection probability.
Last updated
  • stability selection
  • subsampling
  • feature selection
  • high dimensional statistics
  • pfer
Keep connecting the ideas
← Back to the glossaryBack to top ↑