Gradient Boosting

What is gradient boosting?

Gradient boosting builds a strong model as a sum of many weak ones, usually shallow decision trees, added one at a time. Each new tree is fitted to the errors of the ensemble so far: formally, to the negative gradient of the loss function with respect to the current predictions. A learning rate shrinks each tree's contribution so the model improves in small, correctable steps. Jerome Friedman formalized the general procedure around 2001, and modern implementations in the XGBoost and LightGBM style added regularization, histogram-based splitting, and speed that made boosted trees the default strong learner for tabular data.

Traders reach for boosting because trading features are exactly the kind of data it dominates: modest-sized tables of numeric inputs with nonlinear thresholds and interactions. Boosted models often beat random forests in tuned benchmarks. The flip side is that the sequential error-chasing that gives boosting its power also makes it eager to chase noise, and financial labels are mostly noise. On market data the gap between a tuned boosted model and a forest is often smaller than the extra overfitting risk, so conservative settings, strong regularization, and strict train/validation discipline matter more here than in most applications.

How it's calculated

The standard form builds the model stagewise, fitting each tree to the loss gradient of the current ensemble.

XGBoost-style variants add second-order gradient information and explicit regularization terms on tree size and leaf weights.

Subsampling rows and columns per round (stochastic gradient boosting) is standard and further reduces overfitting.

How traders use it

• Directional classification: engineered features feed a boosted classifier predicting the label defined by the prediction horizon, with early stopping on a chronologically later validation slice rather than a random one.
• Taming it for markets: practitioners lower depth (2 to 4), lower the learning rate, subsample rows and features, and stop early; default library settings are tuned for cleaner data than markets provide.
• Meta-labeling and filtering: a boosted model predicts whether an existing strategy's signal will succeed, letting the base strategy provide direction while the model gates participation.
• Reading feature importance cautiously: gain-based importances flatter high-cardinality features and split credit erratically among correlated inputs, so permutation importance on held-out data is the safer read.
• Calibrating outputs: boosted probability estimates are often distorted toward the middle, so checking a probability calibration curve before sizing on the probabilities is standard practice.

Gradient boosting vs. neighboring methods