Bagging, short for bootstrap aggregation, fits multiple versions of a prediction model on resampled training data and combines their outputs. In quantitative research, it can make an unstable forecast less sensitive to the particular observations used to fit it. It does not turn one market history into many independent histories.
A deep decision tree is a useful candidate: a small change to its training sample can change its splits substantially. Bagging tries to smooth that instability by averaging fitted models rather than forcing a single tree to be simple. It also applies to learners other than trees. Breiman's original technical report connects the method's benefit to this sensitivity to changes in the learning sample.
One input, three predictions
Start with an eligible training set. In ordinary bagging, each bootstrap sample draws the same number of rows as that set, with replacement: a row can appear repeatedly or be absent. Fit a separate learner to each sample. Their fits can run in parallel because a learner does not need a preceding learner's fitted output.
For regression, evaluate all fitted learners at the same input and average:
Here is the number of fitted learners. Suppose three learners predict the same asset's next-day simple return as 3, 5 and 7 basis points. Their unweighted aggregate is basis points. These are supplied predictions for an arithmetic illustration; no training results or forecasting accuracy are implied. The official regression API likewise defines its prediction as the mean of the learners' predictions.
Original synthetic mechanism and separate variance model. Row samples and predictions are stipulated, not a fitted forecasting experiment. Variances are rounded to two decimal places.
For classification, one aggregation rule is a vote for the most frequent predicted class. Multiclass voting need not produce an absolute majority, and ties need an explicit rule. Some implementations instead average class probabilities; that is a different aggregation convention. Never average numerical codes for class labels as though the codes were measured outcomes.
Averaging leaves shared errors behind
The key quantity is variance across possible fits at a fixed input, rather than the cross-sectional spread of three predictions observed once. In a simplified model, let every learner's prediction have variance , with common pairwise correlation . Then
The expression follows by adding all variances and covariances in the mean. For this illustration, restrict . With individual variance 4 squared basis points and three learners, the ensemble variance is when uncorrelated, at correlation , and at correlation . These are assumptions for a separate variance model; the three supplied predictions above do not estimate them.
As the number of learners grows, the variance approaches . More resamples can reduce the variability from the finite number of bootstrap fits, while shared training history still limits what averaging achieves. Independently generated bootstrap draws, conditional on the observed training set, do not make the learned predictors independent across possible market histories. Averaging also gives no general guarantee of removing bias or improving future trading performance.
A random forest adds feature subsampling at tree splits to encourage further diversity. Boosting instead constructs learners sequentially using the current ensemble. The owned Bagging and Random Forests lesson develops the tree mechanics and a synthetic Python comparison; the full lesson requires membership, with a public preview.
Out of a bag is not out of the future
For a given learner, an out-of-bag row is one excluded from its bootstrap sample. An OOB prediction combines only learners whose fitting samples omit that row. Under suitable sampling assumptions, this provides a convenient internal error estimate.
Financial observations can share underlying returns, overlapping label windows or a market regime. Omitting one row does not remove related information from neighbouring training rows. A shuffled resample can also include observations later than the date being assessed. López de Prado's discussion of observation redundancy specifically warns that financial OOB accuracy can be inflated; scikit-learn's time-series validation guidance also explains why nearby training and test observations undermine ordinary random splits.
Define the forecast cutoff first, fit preprocessing and learners using eligible information, and assess the complete procedure on appropriately separated later data. Where labels overlap, account for their information intervals. Resampling within a training window does not fix look-ahead bias, choose an appropriate cross-validation design or establish that a signal survives costs. The membership lesson Time-Ordered Train, Validation and Test Design, available with a public preview, turns those information boundaries into a fitting schedule.
Sources
- Breiman, Bagging Predictors, Berkeley Technical Report 421, September 1994, §§1 and 4. The linked document is the technical-report version.
- James, Witten, Hastie, Tibshirani and Taylor, An Introduction to Statistical Learning with Applications in Python (2023), §8.2.1–8.2.2, pp.343–347. Author resource site.
- López de Prado, Advances in Financial Machine Learning (2018), §6.3.1 and §6.3.3: forecast variance and observation redundancy.
- scikit-learn, BaggingClassifier, class-probability averaging and voting conventions.
- scikit-learn, BaggingRegressor, prediction aggregation; cross-validation of time series data, temporal dependence and assessment.