Bagging

What is bagging?

Bagging, short for bootstrap aggregating, is an ensemble method that trains many copies of the same model on different bootstrap samples of the training data and averages their predictions (or takes a majority vote for classification). It was introduced by Leo Breiman in 1996. Each bootstrap sample is drawn from the original training set with replacement, so each model sees a slightly different version of history and makes slightly different errors; averaging washes much of that disagreement out.

The method exists to attack variance. A flexible learner such as a deep decision tree fits its particular training sample aggressively, so retraining on a slightly different sample can produce a very different model. Bagging does not make any single model better; it makes the ensemble's output more stable by averaging over the sampling noise. Bias is largely unchanged, which is why bagging helps high-variance learners a lot and stable learners such as heavily regularized linear models very little.

Traders care because financial prediction is exactly the setting where variance dominates: signal-to-noise is low, effective sample sizes are small, and a model refit on a shifted window can flip its conclusions. Bagged models tend to produce steadier signals across refits, and the bootstrap structure yields a useful byproduct, out-of-bag predictions, where each observation is scored only by the models that never trained on it. One caution is specific to markets: standard bootstrap resampling assumes roughly independent observations, and overlapping or autocorrelated financial samples violate that, so block bootstrap variants are often preferred.

How it's calculated

The canonical form averages B models, each trained on an independent bootstrap resample of the training set.

Each bootstrap sample omits roughly 36.8 percent of the original observations on average; those out-of-bag observations provide a built-in validation estimate.

For classification, averaging predicted probabilities is a common alternative to hard voting.

With autocorrelated financial series, block or stationary bootstrap variants resample contiguous chunks to respect serial dependence.

How traders use it

• Stabilizing noisy signal models: bagging a high-variance learner over bootstrap samples of the training window tends to reduce the signal flip-flopping that plagues single models refit on rolling data, at the cost of extra computation.
• Out-of-bag evaluation: the out-of-bag error gives a nearly free estimate of generalization performance, useful when data is too scarce to sacrifice a large holdout, though it does not replace proper walk-forward testing on time-ordered data.
• As the engine inside random forests: bagging plus per-split feature randomness is the random forest recipe, and most practical exposure to bagging in trading comes through that algorithm.
• Guarding against sample sensitivity: comparing predictions across the bagged members reveals how much a conclusion depends on which observations happened to be sampled, a cheap robustness check related in spirit to model overfitting diagnostics.
• Honest limitation: bagging cannot fix a biased or information-free feature set, and averaging many models trained on the same flawed data produces a confident version of the same flaw.

Bagging vs. other ensemble approaches