Skip to content

Extreme gradient boosting

XGBoost combines efficient boosted trees with regularization. See how gradient and Hessian sums determine a leaf score, and why shrinkage gives a different prediction update.

Extreme gradient boosting (XGBoost) is a machine-learning library best known for efficient, regularized gradient-boosted trees. It adds trees sequentially to improve a chosen training objective. For a quant predicting next-day returns from tabular features, it offers nonlinear interactions without requiring a neural network; a better training fit still needs an honest test on later data.
XGBoost is a particular system for boosting, rather than a synonym for every boosting algorithm. The original system combined regularized tree learning with sparsity-aware split search and computational optimizations. Its second-order method builds on earlier boosting research. Chen and Guestrin, §§2–4

What the next leaf is trying to do

A decision tree sends each observation to a leaf with a numerical score. A boosted ensemble adds the scores from successive trees. At each new stage, derivatives of the loss describe which changes would help the current predictions.

For one proposed leaf containing observations II, define

G=∑i∈I∂ℓ(yi,y^i)∂y^i,H=∑i∈I∂2ℓ(yi,y^i)∂y^i2.G=\sum_{i\in I}\frac{\partial\ell(y_i,\hat y_i)}{\partial\hat y_i}, \qquad H=\sum_{i\in I}\frac{\partial^2\ell(y_i,\hat y_i)}{\partial\hat y_i^2}.

Here GG aggregates the local direction and HH the curvature. Under a second-order loss approximation and an L2 penalty λw2/2\lambda w^2/2, the part depending on this leaf's score is

Q(w)=Gw+12(H+λ)w2,w∗=−GH+λ.Q(w)=Gw+\frac12(H+\lambda)w^2, \qquad w^*=-\frac{G}{H+\lambda}.
This fixed-leaf result assumes H+λ>0H+\lambda>0, no L1 penalty and no extra constraint on the leaf score. It does not choose the tree structure. A leaf-count penalty γT\gamma T also matters when comparing structures with different numbers of leaves, although it is constant when optimizing a fixed structure. Official boosted-tree derivation

One leaf, then one cautious update

Take a synthetic next-day-return regression leaf with three observations. Current predictions are zero and targets are 1,2,31,2,3, all in standardized score units. Use the sum of half-squared errors, ℓ(y,y^)=(y−y^)2/2\ell(y,\hat y)=(y-\hat y)^2/2, rather than their mean. Each observation then contributes gi=y^i−yig_i=\hat y_i-y_i and hi=1h_i=1.

Thus G=−6G=-6, H=3H=3 and, choosing λ=1\lambda=1,

w∗=63+1=1.5,Q(1.5)=−6(1.5)+12(4)(1.5)2=−4.5.w^*=\frac{6}{3+1}=1.5, \qquad Q(1.5)=-6(1.5)+\frac12(4)(1.5)^2=-4.5.

The derivative is Q′(w)=−6+4wQ'(w)=-6+4w, so it vanishes at 1.51.5; positive curvature makes this the minimum. Equivalently, Q(w)=2(w−1.5)2−4.5Q(w)=2(w-1.5)^2-4.5.

A negative increment is not a negative loss. The baseline half-squared-error sum is 77. At the unshrunken score 1.51.5, the direct loss plus L2 penalty is 2.52.5: a change of −4.5-4.5. The quadratic expansion is exact for this squared-error example. With a general smooth loss, it is a local approximation. Scaling or averaging the loss changes GG and HH, so the same numerical penalty need not produce the same score.
Two convex leaf-objective curves: without an L2 penalty the minimum is w=2 and increment minus 6; with lambda=1 the minimum is w=1.5 and increment minus 4.5. A learning rate of 0.1 scales 1.5 to an added prediction of 0.15.
Two convex leaf-objective curves: without an L2 penalty the minimum is w=2 and increment minus 6; with lambda=1 the minimum is w=1.5 and increment minus 4.5. A learning rate of 0.1 scales 1.5 to an added prediction of 0.15.
Original synthetic fixed-leaf calculation. Vertical values are objective changes under the stated loss convention, not total loss or trading returns.

After finding the leaf score, a learning rate η=0.1\eta=0.1 gives the prediction update

y^new=y^old+ηw∗=y^old+0.15.\hat y_{\mathrm{new}}=\hat y_{\mathrm{old}}+\eta w^* =\hat y_{\mathrm{old}}+0.15.

The 1.51.5 is the unshrunken optimum; 0.150.15 is the contribution actually added at this learning rate. They answer different questions. The planned prediction-arithmetic illustration uses these supplied values only; it does not train a tree or solve the leaf optimization.

Controls a researcher should distinguish

In the tree booster, eta (alias learning_rate) scales the update, while lambda (alias reg_lambda) penalizes leaf weights. gamma sets the minimum reduction needed for a further split. max_depth limits tree depth, and min_child_weight constrains summed Hessian support in a child; it is not universally an observation count. These controls address different routes to excessive complexity. Official parameter definitions
The Boosting: AdaBoost and Gradient Boosting lesson develops the sequential fitting and shrinkage intuition, with Python examples. Its full text requires membership; a public preview is available.
For financial research, select complexity and the number of stages using a validation period that respects information availability, and reserve later data for evaluation. Tree regularization cannot remove look-ahead bias, rescue a leaked feature, or establish that a forecast survives costs. Raw scores also need an interpretation appropriate to the selected objective: a regression score is not automatically a probability or a tradable return.

Sources

Last updated
  • machine learning
  • boosting
  • decision trees
  • regularization
  • financial validation
Keep connecting the ideas
← Back to the glossaryBack to top ↑