Null feature importance compares a feature's observed importance with scores produced under a deliberately random reference. Shadow features are shuffled copies of input columns used as competing noise controls. Both help a quant ask whether an impressive signal ranking exceeds what the modelling procedure can manufacture from randomness.
A positive score alone does not answer that question. Flexible models can assign importance to noise, and the score's scale depends on the model and importance definition. A random reference supplies context; statistical significance additionally requires a valid test design.
Three shuffles, three different questions
| Construction | What changes? | What does the comparison address? |
|---|---|---|
| Shuffled-target null importance | Permute target labels, refit the model, and recompute each feature's importance across repetitions. | How unusual is this importance under the chosen no-association reference? |
| Shadow features | Shuffle copies of input columns; fit originals and shadows together. | Does an original input outperform deliberately random competitors? |
| Permutation importance | Keep a fitted model fixed and shuffle one input in the evaluation data. | How much does the fitted model's score deteriorate when that input is disrupted? |
The last row is an importance measure, rather than a significance test by itself. It can also be used as the statistic in a separate null-calibration procedure. The scikit-learn explanation makes the fixed-model distinction explicit.
Target shuffling removes the association between the target and all predictors. It therefore does not automatically test whether one feature adds information conditional on the others. Ojala and Garriga show why changing the permutation construction changes the question being tested. State the null before choosing the shuffle.
A shadow preserves its original column's empirical marginal values, but ordinary row shuffling disrupts its alignment with the target and other columns. Boruta is a specific repeated shadow-based selection algorithm; adding one shuffled column and retaining anything above it is not the full procedure. The original Boruta paper, reprinted in Chapter 2 of Kursa's dissertation, explains the noise-reference construction.
Five null scores cannot establish a small tail
Consider a synthetic, pre-specified importance statistic where larger values are more extreme. The observed score is 0.12. Five null randomisations produce 0.02, 0.04, 0.06, 0.08 and 0.10. These are stipulated teaching values, not output from a fitted market model.
Assume the null makes the observed and randomised arrangements exchangeable: under that null, the observed arrangement has no privileged rank. For the standard plus-one randomisation calculation, count null scores at least as large as the observed score:
Here and there are zero exceedances, so
Above every sampled null score does not mean a p-value of zero. In this illustration the observed score occupies the most extreme of six ranks; under exchangeability each rank is equally likely. A value of about 16.7% is not significant at a 5% level. Five null draws give very coarse resolution.
Original stipulated-score example, not fitted financial results. The observed arrangement counts alongside five null draws.
Phipson and Smyth explain the plus-one correction and distinguish permutation-sampling schemes. It is a valid randomisation p-value under the appropriate design, not a universal conversion of importance into significance. A ratio to a null mean, an empirical percentile, or the fraction of shadow scores beaten may be useful screening summaries; calling one a p-value requires its own calibration argument.
What can break the reference in quant research?
Time and panel dependence. Financial labels may overlap in time, and assets may share market shocks. Arbitrary row shuffling can destroy this structure along with predictive association. A block or within-group construction may be appropriate for a particular null, but needs justification; the word “block” alone does not establish exchangeability. Chronological validation and a valid randomisation reference solve different problems. The official time-series guidance explains why treating nearby observations as independent can mislead.
Selection leakage and repeated searches. Choose features within training data, including any tuning of thresholds or importance models. The official leakage example shows how selection on the whole sample contaminates evaluation. If the tested statistic includes a search, the null procedure must account for that search. Testing many candidate signals also creates a multiple-testing problem; an unadjusted result for one feature is not a guarantee for the selected set.
Model dependence and correlated inputs. An input can look weak because the fitted model cannot use it, or because another input substitutes for it. The permutation-importance documentation discusses this masking effect. Neither a weak importance nor a failed noise comparison proves that the underlying economic variable is irrelevant.
For the model mechanics behind common importance scores, read Bagging and Random Forests, available with a public preview and membership for the full lesson. A favourable null comparison remains a screening result: assess the complete forecasting or trading process on later out-of-sample data, including costs where relevant.
Sources
- Phipson, B. and Smyth, G. K. (2010), Permutation p-values should never be zero: calculating exact p-values when permutations are randomly drawn, especially §§2–6: finite randomisation ranks, plus-one calculation and sampling schemes.
- Ojala, M. and Garriga, G. C. (2010), Permutation Tests for Studying Classifier Performance, abstract: label permutations and restricted feature permutations address different null questions.
- Kursa, M. B. and Rudnicki, W. R. (2010), Feature Selection with the Boruta Package, §2, read in the author-hosted dissertation reprint, Chapter 2, printed pages 2–4.
- scikit-learn developers, Permutation feature importance, §§5.2–5.2.3; Common pitfalls, §§12.2–12.2.2; and Cross-validation, §§3.1.2.6–3.1.3. Documentation checked 2026-11-01. Financial randomisation cautions are implications of dependence, not a claim that a specific block scheme is validated.