Boosting builds an ensemble sequentially, adding learners that respond to its current errors. See an additive update, the AdaBoost distinction and why lower training loss does not guarantee better forecasts.
Boosting
Boosting is a family of ensemble learning methods that fits learners sequentially, with each new learner responding to the errors or loss of the ensemble built so far. In quantitative research, it can turn small prediction rules into a flexible return, volatility or classification model. That flexibility also lets it chase noise: adding learners does not guarantee better future forecasts.
The defining feature is the dependence between fitting stages. A new learner is trained using information about the current ensemble, rather than being another independent forecast added to a committee. Small decision trees are common base learners, but boosting is not restricted to trees.
A correction, rather than an average
For a gradient-boosting regression model, write a stage update as
Fm(x)=Fm−1(x)+ηhm(x),
where Fm−1(x) is the current prediction, hm(x) is the new learner's output and η is the learning rate. Here hm includes any fitted step or leaf values before learning-rate shrinkage. An initial constant prediction F0 usually starts the sequence.
Suppose a synthetic model predicts a next-period return of 4 basis points for one feature vector. A new learner supplies a correction of 2 basis points before shrinkage. With a learning rate of 0.5, the updated prediction is
4+0.5(2)=5 basis points.
It is not the arithmetic average of 4 and 2, which would be 3. The learner's 2 is an adjustment to the current ensemble, not a second standalone estimate of the return.
Synthetic boosting update in basis points: prediction 4 plus learning rate 0.5 times learner output 2 gives 5. For target 6, the residual falls from 2 to 1 and half-squared loss from 2 to 0.5 basis points squared.
Original synthetic regression example with a supplied learner output; no model is fitted. The new learner contributes a correction in basis points. Loss is one half of the squared residual, in basis points squared.
If this training observation's target is 6 basis points, its residual falls from 2 to 1 basis point. For the explicitly chosen loss L(y,F)=21(y−F)2, its loss falls from 2 to 0.5 basis points squared. These are the arithmetic consequences of a supplied update, not evidence that a fitted model will produce that correction.
What does the next learner learn?
Gradient boosting fits a learner to the negative gradient of a chosen loss with respect to the current predictions, then adds a suitably scaled update. For half-squared-error loss,
−∂F(xi)∂L(yi,F(xi))=yi−F(xi).
That is the ordinary residual. After the update, residuals must be recomputed before fitting the next learner. For other losses, the fitting targets are pseudo-residuals and need not equal observed minus predicted values. Classification may build an additive score that is then transformed into a probability; arbitrary additive scores are not automatically probabilities.
AdaBoost, in its classic binary classification form, instead reweights training observations so mistakes made by the current learner receive greater relative influence in the next fit. It combines the resulting classifiers using a weighted vote. AdaBoost and gradient boosting share sequential correction, but “fit the residuals” is not a complete description of every boosting algorithm. Chapter 10 of The Elements of Statistical Learning develops their common additive-model perspective.
Bagging fits models on resampled data and aggregates predictions; it does not require each new model to correct the current aggregate. Sampling alone is therefore a poor way to distinguish them: stochastic gradient boosting also samples observations at each stage, commonly without replacement. The fitting dependency is what matters.
For a fuller derivation and code, Boosting: AdaBoost and Gradient Boosting covers both mechanisms and shrinkage. The full lesson requires membership; its public preview introduces the scope.
Training improvement is not a trading result
The learning rate, number of stages and learner complexity interact. Smaller updates generally need more stages; neither a small learning rate nor shallow trees prevents overfitting. These controls must be selected using appropriate validation, rather than by the training loss alone.
Even the toy update can disappoint on new data. If a separate observation with the same feature values realizes 3 basis points, changing its forecast from 4 to 5 raises its half-squared error from 0.5 to 2 basis points squared. A correction that helps a training observation can hurt a future one.
For financial forecasts, preserve the time ordering of information, prevent look-ahead bias and account for overlapping label periods when constructing validation splits. Choose tuning settings on validation data, then evaluate on a separate holdout. A lower prediction loss is also distinct from profitable trading after costs: boosting supplies a model, not a guarantee of an exploitable signal.
