Stepwise and best-subset selection: Definition & Examples

Stepwise selection builds a regression model through local additions or deletions of predictors; best-subset selection seeks the best predictor combination at a specified size. Both are forms of feature selection. A quant choosing return predictors can use them to reduce the input set, but a faster search may miss a better combination, and a better training fit may forecast worse.

Two decisions must stay separate: which subset is best at a given size, and which size to use.

Search the columns, then choose the size

For a set of selected columns

S

, refit ordinary least squares and compute its training residual sum of squares:

\operatorname{RSS}(S)=\min_{b,\,\beta_S}\sum_{i=1}^n\left(y_i-b-\sum_{j\in S}x_{ij}\beta_j\right)^2.

Here $b$ is an intercept included in every candidate. If the declared model omits an intercept, fix $b=0$ throughout. Keep the observations and fitting convention the same when comparing candidates.

At size $k$ , best-subset selection solves

S_k^*\in\operatorname*{arg\,min}_{|S|=k}\operatorname{RSS}(S).

The winning subsets for successive sizes need not be nested. Direct enumeration compares $\binom{p}{k}$ subsets at size $k$ , or $2^p$ across all sizes including the empty subset. Exact search shortcuts can avoid some fits; “best” requires reaching or certifying the optimum for the declared objective.

Search	Candidate move	What it can miss
Forward stepwise	Start with no selectable predictors. Try each remaining column, refit the whole candidate model, and add the one with the lowest RSS.	A previously selected predictor cannot leave this path.
Backward stepwise	Start with all candidates. Try deleting each retained column, refit, and keep the deletion with the smallest resulting RSS.	A removed predictor cannot return to this path.
Bidirectional stepwise	Permit additions and removals under declared entry, removal and stopping rules.	Local moves still need not find the global best subset.

These are search conventions, not a single universal implementation. Bidirectional procedures may use significance thresholds or a penalised criterion. Other sequential selectors score each proposed move by cross-validation, as in the official scikit-learn implementation. Specify the score, direction, tie handling and stopping rule; an RSS-based path and a validation-based path can differ.

The ISLP authors’ treatment, §§6.1.1–6.1.3, separates fixed-size search from choosing among sizes. A complete forward RSS path uses

1+p(p+1)/2

fits including the starting model. With

p=20

, that is 211 fits, compared with 1,048,576 subsets in direct exhaustive search. These counts describe computation, not a number of independent hypothesis tests or a complete adjustment for adaptive search.

A better first step can lead to a worse pair

Take this synthetic, three-observation regression with no intercept. The values have arbitrary response and feature units; they are not financial returns.

Observation	$x_1$	$x_2$	$x_3$	$y$
1	1	1	0	1
2	1	0	1	1
3	0.5	0	0	0

With one feature, the RSS values are

\operatorname{RSS}(\{1\})=\frac{2}{9},\qquad \operatorname{RSS}(\{2\})=\operatorname{RSS}(\{3\})=1.

For $x_1$ alone, the fitted coefficient is $8/9$ and the residual vector is $(1/9,1/9,-4/9)^\top$ , giving $2/9$ . Forward selection therefore chooses $x_1$ first.

At the next step it compares only two pairs:

\operatorname{RSS}(\{1,2\})=\operatorname{RSS}(\{1,3\})=\frac15=0.2.

For example, refitting $x_1,x_2$ gives coefficients $(4/5,1/5)$ and residuals $(0,1/5,-2/5)^\top$ . Either tie-breaking choice leaves RSS $0.2$ .

But there are three possible pairs. The excluded pair satisfies

y=x_2+x_3

, so

\operatorname{RSS}(\{2,3\})=0.

Enumeration of all three pairs proves the gap: the best pair excludes the best single predictor. Forward selection cannot reach it at size two because it retains $x_1$ .

Synthetic no-intercept regression with three observations: x1 alone has RSS 2/9, while x2 and x3 alone each have RSS 1. Forward selection retains x1, so either reachable pair has RSS 0.2. The pair x2 plus x3 has RSS zero and is the best of all three pairs.

Original synthetic example. Each subset is refitted by least squares without an intercept. The best single feature need not belong to the best pair; these are training errors only.

Backward selection happens to find the best pair here by removing $x_1$ from the full fit. That is a property of this example, not a guarantee for backward selection. The full three-column model is uniquely estimable here, but saturated: three observations leave no residual degrees of freedom for estimating error variance. The example establishes a search failure, not valid coefficient tests or future performance.

Training RSS cannot choose the winner across sizes

Adding a column preserves the old fit as an available choice by setting its new coefficient to zero. Consequently, minimum training RSS cannot increase along a nested forward path. The exhaustive best-at-each-size RSS also cannot increase, even though its selected subsets may change. This mechanical improvement is no evidence of better test predictions.

Use validation to choose size, or a declared criterion such as AIC or BIC. For likelihood-based criteria, count the estimated likelihood parameters consistently: slopes, an intercept if fitted, and an unknown error variance in a Gaussian model. The official R likelihood documentation makes the estimated-scale convention explicit. A size penalty does not automatically account for every research choice or establish predictive accuracy. Do not apply ordinary Gaussian likelihood comparisons to this saturated exact-fit toy as if it supplied an estimated positive noise variance.

Backward selection also needs a well-defined starting fit. Classical coefficient-based versions require identifiable columns; adding an intercept uses another column. A generalised-inverse fit can produce fitted values in a rank-deficient model, but does not restore unique coefficients or ordinary full-model significance tests. Strong multicollinearity can make selected names unstable even without exact dependence.

Least Squares and Regression Geometry develops the projection argument behind RSS monotonicity. Its public preview is available; the full lesson requires membership.

Assess the whole search

A valid held-out score must assess the selector as well as the final coefficients. For an RSS-based search, rebuild the entire path on each training fold, then score its candidates on that fold’s validation observations. Selecting columns once using all observations and cross-validating only the final regression leaks the held-out outcomes into the selection. The official leakage example demonstrates why this can create apparent predictive power even with random targets.

If size, search direction, scoring rule or other settings are chosen from validation results, assess that chosen procedure on an outer held-out set or a separate final test. Each inner split must refit all learned preprocessing and selection using only its own training data. A selector that internally uses cross-validation still needs outer assessment: its internal score helped make the choice.

For return forecasting, preserve chronology in these splits and fit only with features and completed targets available at the historical decision time. Overlapping outcome intervals may require purging. Freeze the candidate universe, search budget, metric and refitting rule before final testing. Time-Ordered Train, Validation and Test Design explains those information boundaries; its full teaching requires membership and has a public preview.

Finally, a small ordinary regression p-value after searching the same responses does not have its usual fixed-model interpretation. Refitting the selected subset by OLS does not undo the search. Independent inference data or a justified selection-aware method is needed for that claim. Tibshirani and co-authors, §1, demonstrate the problem and develop conditional tests under a specified Gaussian observation model. Those assumptions and the selection event matter; the procedure is not a universal correction for financial time series.

Sources

James, G., Witten, D., Hastie, T., Tibshirani, R. and Taylor, J. (2023), An Introduction to Statistical Learning with Applications in Python, §§6.1.1–6.1.3, pp. 231–239. Local textbook text consulted; book and edition information available from the authors’ official site.
scikit-learn developers, SequentialFeatureSelector, class description and parameters direction, scoring and cv. Official documentation checked 30 October 2026; its move score uses cross-validation rather than the training RSS used in the toy.
scikit-learn developers, Common pitfalls: data leakage, §§12.2–12.2.2. Training-only transformations, feature selection and held-out assessment.
Tibshirani, R. J., Taylor, J., Lockhart, R. and Tibshirani, R. (2016), Exact Post-Selection Inference for Sequential Regression Procedures, Journal of the American Statistical Association 111(514), 600–620, §1. Same-data selection and naive coefficient tests; model assumptions for conditional inference. The numerical counterexample and figure here are original.
R Core Team, Extract Log-Likelihood, Details and Value. Official documentation checked 30 October 2026 for estimated scale and likelihood parameter counting.

Search the columns, then choose the size

A better first step can lead to a worse pair

Training RSS cannot choose the winner across sizes

Assess the whole search

Sources

Related terms

Feature selection

Ordinary least squares

P-value

Residual sum of squares