Model Stacking

What is model stacking?

Model stacking, also called stacked generalization, combines several predictive models by training a second-level model on their outputs. A set of base learners, for example a random forest, a gradient boosting model, and a linear classifier, each produce a prediction for the same target; a meta-learner then takes those predictions as inputs and learns how to blend them into a final output. Unlike simple averaging or ensemble voting, the combiner can learn that one base model deserves more weight overall, or more weight in particular conditions.

The idea was introduced by David Wolpert in 1992 under the name stacked generalization. The critical mechanical detail is that the meta-learner must be trained on out-of-fold predictions: each base model predicts rows it never trained on, and only those predictions feed the second level. If the meta-learner instead sees in-sample base predictions, it learns to trust models exactly where they memorized the data, which is a direct route to model overfitting.

Traders care because market models built from different logic, trend-following features versus mean-reversion features versus volatility features, tend to fail at different times. A stack lets the second level exploit that complementarity rather than forcing a fixed blending rule. The cost is an extra layer of complexity and an extra layer of things that can silently leak future information, so the validation burden is heavier than for any single model.

How traders use it

• Blending heterogeneous signal models: base learners trained on distinct feature groups (momentum, volume, volatility) feed a small meta-learner, often logistic regression, that outputs one probability per bar.
• Combining timeframes: base models predict the same label from different lookback horizons, and the stack learns which horizon carries weight in the current environment.
• Regularizing the combiner: practitioners usually keep the meta-learner deliberately simple, since a flexible second level can overfit the base predictions, which are already correlated with each other.
• Diagnostics: if the stack barely beats its best base model out of sample, the base models are probably making the same errors; diversity of inputs and model families matters more than adding layers.
• Validation: the whole pipeline, base models plus meta-learner, must be evaluated with walk-forward analysis as one unit, since tuning any layer against the test period contaminates the result.

Model stacking vs. related ensemble ideas