# Bagging A Machine Learning concept (Ensembles) in the LuxAlgo Library, with 1 indicator implementation. ## What is bagging? Bagging, short for bootstrap aggregating, is an ensemble method that trains many copies of the same model on different bootstrap samples of the training data and averages their predictions (or takes a majority vote for classification). It was introduced by Leo Breiman in 1996. Each bootstrap sample is drawn from the original training set with replacement, so each model sees a slightly different version of history and makes slightly different errors; averaging washes much of that disagreement out. The method exists to attack variance. A flexible learner such as a deep [decision tree](https://www.luxalgo.com/library/concept/decision-trees/) fits its particular training sample aggressively, so retraining on a slightly different sample can produce a very different model. Bagging does not make any single model better; it makes the ensemble's output more stable by averaging over the sampling noise. Bias is largely unchanged, which is why bagging helps high-variance learners a lot and stable learners such as heavily regularized linear models very little. Traders care because financial prediction is exactly the setting where variance dominates: signal-to-noise is low, effective sample sizes are small, and a model refit on a shifted window can flip its conclusions. Bagged models tend to produce steadier signals across refits, and the bootstrap structure yields a useful byproduct, out-of-bag predictions, where each observation is scored only by the models that never trained on it. One caution is specific to markets: standard bootstrap resampling assumes roughly independent observations, and overlapping or autocorrelated financial samples violate that, so block bootstrap variants are often preferred. ## How it's calculated The canonical form averages B models, each trained on an independent bootstrap resample of the training set. ``` D_b = bootstrap sample of D, drawn with replacement, size n (for b = 1..B) f_b = model trained on D_b regression: f_bag(x) = (1/B) * sum over b of f_b(x) classification: f_bag(x) = majority vote of f_1(x), ..., f_B(x) D: original training set of n observations B: number of bootstrap replicates (commonly 50 to several hundred) f_b: the b-th base model, all sharing one learning algorithm f_bag: the aggregated ensemble prediction ``` Each bootstrap sample omits roughly 36.8 percent of the original observations on average; those out-of-bag observations provide a built-in validation estimate. For classification, averaging predicted probabilities is a common alternative to hard voting. With autocorrelated financial series, block or stationary bootstrap variants resample contiguous chunks to respect serial dependence. ## How traders use it - Stabilizing noisy signal models: bagging a high-variance learner over bootstrap samples of the training window tends to reduce the signal flip-flopping that plagues single models refit on rolling data, at the cost of extra computation. - Out-of-bag evaluation: the out-of-bag error gives a nearly free estimate of generalization performance, useful when data is too scarce to sacrifice a large holdout, though it does not replace proper walk-forward testing on time-ordered data. - As the engine inside random forests: bagging plus per-split feature randomness is the [random forest](https://www.luxalgo.com/library/concept/random-forest/) recipe, and most practical exposure to bagging in trading comes through that algorithm. - Guarding against sample sensitivity: comparing predictions across the bagged members reveals how much a conclusion depends on which observations happened to be sampled, a cheap robustness check related in spirit to [model overfitting](https://www.luxalgo.com/library/concept/model-overfitting/) diagnostics. - Honest limitation: bagging cannot fix a biased or information-free feature set, and averaging many models trained on the same flawed data produces a confident version of the same flaw. ## Bagging vs. other ensemble approaches - **Random Forest** (https://www.luxalgo.com/library/concept/random-forest/): A random forest is bagging applied to trees with an extra decorrelation trick, random feature subsets at each split; plain bagging uses the full feature set in every model. - **Gradient Boosting** (https://www.luxalgo.com/library/concept/gradient-boosting/): Boosting trains models sequentially, each correcting the last, and mainly reduces bias; bagging trains models independently in parallel and mainly reduces variance. - **Model stacking** (https://www.luxalgo.com/library/concept/model-stacking/): Stacking combines different model types through a trained meta-learner; bagging combines many instances of one model type through simple averaging or voting. - **Ensemble voting of signals** (https://www.luxalgo.com/library/concept/ensemble-voting-of-signals/): Signal voting aggregates heterogeneous trading rules at the decision level; bagging is a statistical procedure that manufactures its ensemble from one learner and resampled data. ## FAQ ### When does bagging actually help? When the base learner is unstable, meaning small changes in training data change its predictions materially. Deep decision trees are the classic beneficiary; stable, heavily regularized models gain little. ### How many models should be bagged? Performance typically plateaus somewhere in the tens to hundreds of replicates, and adding more mostly costs computation. Monitoring out-of-bag error as B grows shows where the curve flattens. ### Is the ordinary bootstrap valid for time series? Not strictly, because it destroys serial dependence. Block bootstrap variants, which resample contiguous segments, are the standard adaptation for autocorrelated financial data. ### Does bagging prevent overfitting? It reduces the variance component of overfitting but leaves bias and data-snooping problems untouched. A bagged model validated on the training period can still fail out of sample. ## Implementations in the Library - Bagging (LuxAlgo): https://www.luxalgo.com/library/indicator/bagging/ ## Related concepts - Ensemble Voting of Signals: https://www.luxalgo.com/library/concept/ensemble-voting-of-signals/ - Model Stacking: https://www.luxalgo.com/library/concept/model-stacking/ --- Source: https://www.luxalgo.com/library/concept/bagging/ (LuxAlgo Library, the encyclopedia of trading & technical analysis). Free to use with attribution: https://www.luxalgo.com/library/license/