Skip to content

Stacking and blending

Stacking learns how to combine models from held-out predictions. Distinguish fixed blends, learned holdout blending and chronological stacking for financial forecasts.

Stacking combines prediction models by training another model—the meta-learner—on their outputs. Its training inputs must be predictions made without fitting the base models on the corresponding outcomes. For a quant forecasting next-day returns or return direction, this creates a second learning problem: which combination of forecasts remains useful on genuinely unseen data?
Blending is used less consistently. It can mean a fixed weighted average, or a learned combiner trained on one separate holdout set rather than cross-validated predictions. H2O's official documentation uses a separate blending_frame to generate the meta-learner's training inputs. State the construction, because the word alone does not tell a reader whether weights were fitted.
Unlike bagging, stacking need not create its base models through bootstrap resampling. Unlike boosting, it need not fit each base model to correct a preceding model. A linear model, a tree model and a neural network can each produce a forecast; a separate model learns how to combine them.

The row that the base model did not learn

Suppose base models f1,…,fMf_1,\ldots,f_M predict the same target, at the same horizon, for row ii. Stacking constructs an input vector

zi=(f^1(−i)(xi),…,f^M(−i)(xi)).z_i=\left(\hat f_1^{(-i)}(x_i),\ldots,\hat f_M^{(-i)}(x_i)\right).

The superscript means that the fit producing this row's prediction excluded its outcome; it does not require literal leave-one-out fitting. In ordinary fold-based stacking, each base model is fitted on the training part and predicts the held-out part. Align every model's predictions to the same row IDs, outcomes and class definitions. The meta-learner gg then learns from pairs (zi,yi)(z_i,y_i).

That exclusion matters. A flexible base model can nearly memorise its training labels. If its fitted values are used as the second model's features, the combiner learns to trust a level of accuracy that may disappear on new observations. The Ting and Witten research paper, §§1–2, describes the held-out prediction dataset and the subsequent full-data base fits. In classification, probability outputs retain distinctions that hard class labels discard.

For inference, refit the base models on the eligible development data under the chosen procedure, then feed their new predictions into the fitted combiner:

y^(x)=g^ ⁣(f^1(x),…,f^M(x)).\hat y(x)=\hat g\!\left(\hat f_1(x),\ldots,\hat f_M(x)\right).

Do not replace the stored held-out training features with these refitted models' in-sample predictions and retrain the combiner on them. The refitted models can also have a different prediction distribution from their smaller training-window versions; assessment must include this refitting step.

A 60/40 calculation is not evidence of learning

Consider one fictional asset-date row. Two supplied out-of-fold predictions for the probability of a positive next-day simple return are 0.20 and 0.70. Fix the weights at 0.60 and 0.40:
pblend=0.60(0.20)+0.40(0.70)=0.40.p_{\mathrm{blend}}=0.60(0.20)+0.40(0.70)=0.40.
The two weighted contributions are 0.12 and 0.28. This demonstrates a fixed blend only. There are no outcomes or fitted coefficients here, so the calculation does not demonstrate learned stacking or show that 60/40 is a useful choice. Being out of fold describes the base predictions' provenance; it does not turn a manually selected average into a trained meta-learner.
A constrained linear combiner could learn nonnegative weights summing to one from many held-out predictions and their realised outcomes. A logistic regression combiner instead learns an intercept and coefficients, then maps its score to a probability. Stacking does not require weights to sum to one, nonnegative coefficients or even a linear combiner. A convex average stays between its input probabilities, but that fact alone does not establish probability calibration.

For market data, held out must also mean available in time

Ordinary random cross-validation can exclude a row while training on its future. That does not recreate a historical forecast. For a chronological stacking design, generate meta-training features by rolling or expanding forward through the development history:

  1. Fit each base pipeline using only information and completed labels available at that cutoff. Fit scalers, feature selection and other learned transformations inside that window too.
  2. Store predictions for the next block. Repeat at later cutoffs, preserving the original row IDs and outcomes.
  3. Once those outcomes have matured, fit the combiner on the stored predictions and labels. The initial warm-up block has no eligible past-only predictions and does not become meta-training data.
  4. Refit bases on eligible development data and assess the frozen complete procedure on a later untouched test block. If live updates are intended, freeze and assess their schedule as well.
A chronological schematic has three prediction folds: fit A to predict B; fit eligible A and B to predict C; fit eligible A through C to predict D. Stored predictions and matured labels from B through D fit the meta-learner. Refit bases on eligible A through D, then pass their predictions on untouched E into the frozen meta-learner.
A chronological schematic has three prediction folds: fit A to predict B; fit eligible A and B to predict C; fit eligible A through C to predict D. Stored predictions and matured labels from B through D fit the meta-learner. Refit bases on eligible A through D, then pass their predictions on untouched E into the frozen meta-learner.
Original schematic: blocks A–D are development history; E is the untouched test. Every fit uses only labels available at its cutoff. Block widths are illustrative; label-completion exclusions at boundaries are not drawn.
With overlapping return labels, a row's date is not enough: its target may finish after the fit cutoff. Account for the outcome intervals, any required purging and operational availability. A fixed gap in row counts is not automatically an adequate rule for variable-length events. The owned lesson Time-Ordered Train, Validation and Test Design develops eligibility, preprocessing and refitting schedules; its preview is public and the full lesson requires membership.
There is a software boundary too. Current scikit-learn stacking uses cross-validated predictions internally; its default folds are not past-only, even with shuffling disabled. Its cross_val_predict contract assigns every input sample to exactly one held-out set. An expanding-window TimeSeriesSplit leaves the initial training block unpredicted, so simply substituting that splitter into the standard stacking fit is not a complete solution. Construct and align the eligible temporal prediction rows explicitly.

A separate chronological blending holdout can be simpler: fit bases on an earlier block and train the combiner on their predictions for a later block after its labels arrive. It leaves fewer observations for combiner fitting, and the holdout is now training data for the ensemble, not its final test. If the bases are then refitted on more history, assess that exact procedure rather than assuming the combiner transfers unchanged.

What earns the extra layer?

Useful base models make complementary errors on eligible held-out observations. Merely giving similar models different names adds little information, while a highly flexible combiner can fit noise in a small meta-training sample. Regularisation, a simple averaging baseline and comparison with the strongest individual base model help establish whether the additional layer is worthwhile.

The combiner's training score is not an independent ensemble score. Selecting base models or repeatedly tuning the stack against the same held-out history also consumes that history for model selection. Preserve later evaluation of the whole procedure. A lower forecasting loss still needs a separate account of turnover, costs and position construction before it implies an improved trading strategy.

Sources

Last updated
  • machine learning
  • ensembles
  • stacking
  • blending
  • temporal validation
Keep connecting the ideas
← Back to the glossaryBack to top ↑