Sure independence screening: Definition & Examples

Sure independence screening (SIS) is a fast predictor filter that ranks variables by their separate associations with a response, with a guarantee of retaining every active variable under specified model and signal conditions. In a quant forecasting pipeline, it can shorten a large candidate signal list before a more expensive joint fit. “Sure” refers to an asymptotic probability statement, not certainty that a particular shortlist contains every useful predictor.

A shortlist from four scores

For the original linear correlation screener, centre the response and standardise each nonconstant predictor using training data. Ranking absolute sample correlations gives the same ordering as ranking the absolute componentwise regression scores, since the response scale is common to all predictors. Here “independence” means each predictor is scored separately; it does not require the predictors to be mutually independent. This is the construction in Fan and Lv, §2.1.

Write the score as

\widehat r_j=\frac{\sum_{i=1}^{n}(x_{ij}-\bar x_j)(y_i-\bar y)}{\sqrt{\sum_{i=1}^{n}(x_{ij}-\bar x_j)^2}\sqrt{\sum_{i=1}^{n}(y_i-\bar y)^2}}.

The formula requires positive sample variances. Keep the

d

largest values of

|\widehat r_j|

, with a declared rule for ties. In a fictional training fold, suppose the absolute scores for features 1–4 are 0.8, 0.7, 0.1 and 0.6. With

d=2

, the shortlist is features 1 and 2. Feature 4 ranks third despite its sizeable marginal association.

Illustrative absolute marginal correlations: feature 1 scores 0.8, feature 2 scores 0.7, feature 3 scores 0.1 and feature 4 scores 0.6. Keeping the largest two retains features 1 and 2; the ranking does not establish their joint coefficients.

Original stipulated screening example. Blue bars and explicit labels identify the retained features; the scores are not fitted market evidence.

These four numbers cannot reveal the best two-variable model. Correlated proxies can take several places in the shortlist, while a variable that helps conditionally can score poorly alone. SIS is a filter within feature selection; a subsequent Lasso regression, for example, fits the survivors jointly.

What survives, and under which model?

For a specified sparse linear data-generating model,

Y=\mathbf X^\top\boldsymbol\beta+\varepsilon, \qquad \mathbb E[\varepsilon\mid\mathbf X]=0, \qquad M_* =\{j:\beta_j\ne0\},

the sure-screening property is

\mathbb P\bigl(M_*\subseteq\widehat M_{d_n}\bigr)\longrightarrow1 \quad\text{as }n\longrightarrow\infty.

Here $p=p_n$ , the support and the retained size may change with sample size. The probability is over training samples from that model. Inclusion permits extra variables: it is weaker than exact support recovery, $\widehat M_{d_n}=M_*$ . It supplies no causal identification.

A simple threshold argument exposes the essential marginal-signal requirement. Let $r_j=\operatorname{Corr}(X_j,Y)$ , let $q_n=\min_{j\in M_*}|r_j|>0$ , and suppose, with probability tending to one,

\max_{1\le j\le p_n}|\widehat r_j-r_j|<q_n/2.

Then every active feature has $|\widehat r_j|>q_n/2$ . A threshold screen at $q_n/2$ retains them all. This is an explanatory sufficient condition, not a prescription using known $q_n$ : the minimum population signal is usually unknown. Nor does it bound the shortlist size. For a top- $d_n$ screen, $d_n$ must also accommodate all active variables and any competing large scores. Even $d_n\ge|M_*|$ alone is insufficient.

Obtaining the uniform error bound requires assumptions about sampling, tails and design as dimension grows. Nonzero joint coefficients alone do not suffice: with predictor covariance $\Sigma$ , the marginal covariance vector is $\Sigma\boldsymbol\beta$ , whose entries can cancel.

One sufficient regime in Fan and Lv, conditions 1–4 and Theorem 1 takes

p_n>n

, uses population-standardised predictors and assumes IID linear-model observations, independent Gaussian errors, whitened-design spherical symmetry and concentration, and

\operatorname{Var}(Y)=O(1)

. It requires

\begin{aligned} \min_{j\in M_*}|\beta_j|&\ge c_2n^{-\kappa}, &\min_{j\in M_*}\left|\frac{\operatorname{Cov}(Y,X_j)}{\beta_j}\right|&\ge c_3,\\ \lambda_{\max}(\Sigma)&\le c_4n^\tau, &\log p_n&=O(n^\xi), \end{aligned}

with positive constants, $\kappa,\tau\ge0$ , $0<\xi<1-2\kappa$ and $2\kappa+\tau<1$ . Under these sufficient conditions, an appropriately chosen $d_n<n$ gives the inclusion property. They are not claimed to be the weakest possible assumptions. Merely having $p>n$ establishes none of them. Dependent financial observations require a justification suited to that dependence.

A joint signal invisible to correlation

Consider an original population example with independent $X_1,X_2\sim\mathcal N(0,1)$ and

Y=X_1X_2.

Both marginal covariances vanish:

\begin{aligned} \operatorname{Cov}(Y,X_1)&=\mathbb E[X_1^2X_2] =\mathbb E[X_1^2]\mathbb E[X_2]=1\times0=0,\\ \operatorname{Cov}(Y,X_2)&=\mathbb E[X_1X_2^2] =\mathbb E[X_1]\mathbb E[X_2^2]=0\times1=0. \end{aligned}

Yet observing both inputs determines $Y$ exactly. Their product has variance 1; a predictor given the product makes zero squared prediction error. Screening only the two raw columns by linear correlation provides no population signal to rank. Finite-sample scores need not equal zero, but that random variation is no reliable rescue.

This nonlinear interaction lies outside the original linear SIS guarantee. It does not refute that theorem. Adding the product as a candidate changes the feature universe: its correlation with

Y

is 1. Interaction construction itself must be justified and assessed within training data. Also, zero correlation does not establish independence: here the conditional variance of

Y

given

X_1

X_1^2

Fit the filter inside the training window

Compute scaling, marginal scores and the retained mask separately inside every training fold. Apply that fitted mask to its later validation block. If choosing

d

or comparing screening variants, repeat the whole fitting process inside inner training splits; keep the final test outside those decisions. Selecting once on all targets leaks held-out outcomes even if the eventual regression is refitted. The official leakage example demonstrates this problem with feature selection.

For return labels, only use outcomes completed and available at the fit cutoff. Time-Ordered Train, Validation and Test Design develops these eligibility and nested-fitting rules; its preview is public and the full lesson requires membership. A retained signal still needs evaluation for predictive usefulness, costs and stability.

Sources

Fan, J. and Lv, J. (2008), Sure independence screening for ultrahigh dimensional feature space, JRSS B 70(5), 849–911. §2.1 defines the correlation screener and inclusion property; §§5.1–5.2 state the sufficient conditions and theorem. The threshold implication, four-score example and interaction calculation above are original explanations.
scikit-learn developers, Common pitfalls and recommended practices, §§12.2.1–12.2.2. Training-only preprocessing, feature selection and cross-validation pipelines.

A shortlist from four scores

What survives, and under which model?

A joint signal invisible to correlation

Fit the filter inside the training window

Sources

Related terms

Correlation

Feature selection