Minimum redundancy maximum relevance (mRMR) is a feature-selection approach that rewards association with the target while penalising overlap with inputs already selected. In a return-forecasting model, several lookback signals may tell much the same story. A slightly weaker signal on its own can therefore be preferred when it brings less redundant information.
It is a feature-selection filter: it scores candidate columns before fitting the final predictor. The classical formulation uses mutual information, which can detect dependence beyond linear correlation. It does not identify causal drivers or promise better trading results.
The next feature must earn its place
Let be the target and the nonempty set of features already chosen. The mutual-information difference (MID) rule selects the remaining candidate with the largest score
The first term measures relevance to the target; the second is average pairwise redundancy with selected inputs. Start with the feature having greatest target relevance, then recompute scores after each addition. Peng, Long and Ding (2005), §2.2, give this incremental rule. Greedy additions do not guarantee the globally best predictive subset.
The author’s FAQ, Q1.2 and Q2.5, distinguishes the variants. MIQ divides relevance by average redundancy instead of subtracting it. These are different criteria; a ratio needs an explicit policy for a zero or near-zero denominator. A reported “mRMR ranking” should identify the variant and estimator.
Why B beats A
Suppose one feature has already been selected. The following are synthetic, already estimated mutual-information values in bits, supplied to illustrate scoring; they are not measurements from market data.
| Candidate | Relevance to target | Redundancy with Z | MID score |
|---|---|---|---|
| A | 0.50 bits | 0.40 bits | 0.10 bits |
| B | 0.40 bits | 0.10 bits | 0.30 bits |
Relevance alone chooses A. MID chooses B: subtracting 0.10 from 0.40 leaves 0.30, whereas A keeps only 0.10 after its larger penalty. These scores are ranking criteria, not predicted returns or forecast-accuracy improvements. For the same inputs, MIQ gives A a ratio of 1.25 and B a ratio of 4; it happens to choose B too, despite using a different rule.
Original synthetic scoring example. Solid blue is the MID remainder; the patterned tail is the redundancy deducted from the full relevance bar.
A penalty is not conditional information
Do not read the difference as “the exact information added after conditioning on Z.” For one selected feature, the identity is
MID omits the last term. Pairwise scores can miss useful interactions. Consider two independent fair bits, and , and a target that is 1 exactly when the bits differ. Each input alone has zero mutual information with the target, but together they determine it completely: one bit of information. Once Z is known, X provides that entire bit; its MID score is still zero. A zero score therefore need not mean zero conditional usefulness.
Estimate inside the research boundary
Estimated MI depends on the sample and estimation choices. Binning a continuous signal changes the variable being analysed; bin boundaries learned from data belong inside training. With nearest-neighbour estimates, neighbourhood size trades estimation variance against bias. Specify whether variables are discrete or continuous and keep information units consistent.
For example, scikit-learn's current
mutual_info_classif documentation reports nats, not bits, and describes a univariate relevance estimator—not a complete mRMR selector. Do not subtract a redundancy estimate in bits from relevance in nats.Fit the selector separately within every chronological training fold, using only features and completed target outcomes available at that fit time. Any choice of subset size, estimator settings or MID versus MIQ belongs inside development validation, including inner folds when tuning. Selecting columns on the full history before cross-validation leaks held-out information; the official preprocessing guidance demonstrates this failure.
Time-Ordered Train, Validation and Test Design explains eligible labels, refitting schedules and how to keep the entire learned pipeline within its window. Its preview is public; the full lesson requires membership. Assess the resulting predictor on later data, because a compact ranking can still be unstable across market regimes or discard interactions the final model could use.
Sources
- Peng, H., mRMR FAQ, Q1.1–Q1.2 and Q2.5. Original author documentation on the filter framework and MID/MIQ score meanings.
- Peng, H., Long, F. and Ding, C. (2005), Feature Selection Based on Mutual Information: Criteria of Max-Dependency, Max-Relevance, and Min-Redundancy, IEEE TPAMI 27(8), 1226–1238. §2.2, equations 4–7, gives the incremental difference criterion; §6 discusses quotient combinations.
- scikit-learn developers,
mutual_info_classif, current stable API documentation. Estimator settings, discrete/continuous treatment and nat units; consulted 1 November 2026. - scikit-learn developers, Common pitfalls: data leakage during preprocessing, §12.2.2. Selection must be fitted on the appropriate training data.