Skip to content

Gradient boosting

Gradient boosting builds an additive prediction function by fitting learners to loss gradients. Follow a shrinkage step, distinguish residuals from fitted corrections, and validate financial forecasts.

Gradient boosting builds a prediction function in stages by fitting each new learner to the negative gradient of a chosen loss at the current predictions. In quant research, this lets a collection of small rules model nonlinear relationships in returns, volatility or default risk. The same flexibility can fit noise, so the loss used to train the model and the procedure used to validate it both matter.
It is a particular form of boosting: the next learner depends on the ensemble built so far. The word gradient identifies what guides the next correction.

Descent in predictions, then a rule for new inputs

For training pairs (xi,yi)(x_i,y_i), let Fm−1(x)F_{m-1}(x) be the current prediction function. At stage mm, calculate

rim=−∂L(yi,F)∂F∣F=Fm−1(xi).r_{im}=-\left.\frac{\partial L(y_i,F)}{\partial F}\right|_{F=F_{m-1}(x_i)}.
These are pseudo-residuals. They describe a descent direction in the vector of fitted training predictions. This is functional gradient descent: the object being improved is the prediction function, rather than just a fixed vector of model coefficients.
The gradient values alone exist only at the observed inputs. Fit a learner, often a small decision tree, to approximate those values as a function of xx. The fitted rule can then supply corrections for inputs outside the training set. It need not reproduce every pseudo-residual exactly. For example, a least-squares leaf containing residuals of −2-2 and 66 predicts their mean, 22, for both rows.

A convenient update notation is

Fm(x)=Fm−1(x)+ηhm(x),F_m(x)=F_{m-1}(x)+\eta h_m(x),
where η\eta is the learning rate and hmh_m includes any fitted step size or loss-specific leaf values before shrinkage. The initial function F0F_0 is often a loss-minimizing constant; for unweighted squared-error regression it is the training target mean.
There are two distinct decisions here. Fitting the tree's regions approximates a useful direction. Choosing a step size, or optimizing each leaf's value against the original loss, sets its scale. Applying η\eta then shrinks that fitted update. For losses other than squared error, a tree fitted to pseudo-residuals may require different leaf values before it is added. The Elements of Statistical Learning, Algorithm 10.3, makes this distinction explicit.

One training row through one stage

Use the half-squared-error convention
L(y,F)=12(y−F)2,−∂L∂F=y−F.L(y,F)=\frac12(y-F)^2, \qquad -\frac{\partial L}{\partial F}=y-F.
Under this convention, the pseudo-residual is the ordinary residual. If the loss were defined without the half, its negative gradient would be 2(y−F)2(y-F) instead. Stating the convention avoids silently mixing gradients and step sizes.
Take one hypothetical training row whose next-period simple-return target is 10 basis points. Its current prediction is 4 basis points, so its negative gradient is 6 basis points. Stipulate that the new learner outputs 6 basis points at this input before shrinkage. This is a supplied output, not the result of a model fit. With η=0.5\eta=0.5,
Fm(x)=4+0.5(6)=7 bp.F_m(x)=4+0.5(6)=7\text{ bp}.
The learner contributes 3 basis points; it does not replace the forecast with 6. The residual becomes 3 basis points, and this row's loss changes from
12(10−4)2=18to12(10−7)2=4.5 bp2.\frac12(10-4)^2=18 \quad\text{to}\quad \frac12(10-7)^2=4.5\text{ bp}^2.
Synthetic half-squared training loss versus prediction in basis points. Target 10 and prediction 4 give negative gradient 6. A stipulated learner output 6 with learning rate 0.5 moves the prediction to 7 and reduces loss from 18 to 4.5 basis points squared.
Synthetic half-squared training loss versus prediction in basis points. Target 10 and prediction 4 give negative gradient 6. A stipulated learner output 6 with learning rate 0.5 moves the prediction to 7 and reduces loss from 18 to 4.5 basis points squared.
Original hypothetical training-row illustration. The curve is L(10,F) = one half of (10 − F) squared. The learner output is supplied, not estimated; lower loss on this row does not establish better held-out forecasts.

After this update, recompute pseudo-residuals before fitting the next learner. Repeatedly fitting the original residuals would ignore what the ensemble has already learned. The same construction extends beyond squared-error regression: with classification losses, the additive quantity may be a score transformed into a probability, rather than a return forecast. “Fit observed minus predicted” is therefore a special case, not the general definition.

The planned ensemble prediction lab can illustrate the supplied additive arithmetic above. It does not fit learners, compute gradients or evaluate forecast quality; the derivation and figure provide those parts of the explanation here.

Shrinkage changes the path, not the need for validation

Smaller learning rates usually require more stages to reach a comparable training loss. Tree depth, minimum leaf size, number of stages and learning rate jointly control model flexibility. Regularisation and early stopping can limit overfitting; neither makes a future improvement automatic.
Even this exact update has no held-out guarantee. A separate observation with the same features could realize a return of 4 basis points. Moving its forecast from 4 to 7 raises its half-squared loss from 0 to 4.5 basis points squared, despite helping the training row.
For financial forecasts, select model settings on validation periods that respect information availability. Fit preprocessing within each training split, avoid look-ahead bias, and account for overlapping outcome intervals when excluding training labels. A chronological split alone does not ensure that every training label was already observable at the forecast time. Keep the final test period separate from the tuning process, and assess any trading rule after costs as well as its prediction loss.
For the fitting mechanism and code, Boosting: AdaBoost and Gradient Boosting develops residual updates and shrinkage. For the financial split design, Finance-aware validation works through purging and an embargo using an exclusion ledger. Both full lessons require membership and offer public previews.

Sources

Last updated
  • gradient boosting
  • machine learning
  • loss functions
  • shrinkage
  • financial machine learning
Keep connecting the ideas
← Back to the glossaryBack to top ↑