Multiple-testing Correction

What is a multiple-testing correction?

A multiple-testing correction is any statistical adjustment that accounts for how many strategies, parameters, or signals were tried before the reported result was selected. Test one rule at the conventional 5% significance level and a false positive is unlikely; test two hundred rules and roughly ten will pass by chance alone, so the best of them can look spectacular while meaning nothing. Corrections restore honesty by raising the evidence bar in proportion to the size of the search.

The problem, known in finance as data snooping or data-mining bias, is arguably the central statistical hazard of backtesting, because modern tooling makes it trivial to test thousands of variants against one shared history. Classical corrections such as Bonferroni and Sidak shrink the per-test significance level as the number of tests grows. Finance-specific machinery goes further: White's reality check, a bootstrap test introduced by Halbert White in 2000, evaluates whether the best rule in a family genuinely beats a benchmark once the full search is accounted for, and Hansen's SPA (superior predictive ability) test refines it to be less distorted by the inclusion of very poor rules in the family.

Traders care because uncorrected backtest selection is one of the main ways illusory edges are manufactured. The practical difficulty is bookkeeping: corrections require knowing how many tests were run, including informal ones, and trials on the same data are correlated, so the effective number of independent tests is smaller than the raw count and must be estimated rather than assumed.

How it's calculated

The classical corrections adjust the per-test significance level alpha for N tests. Bootstrap methods such as the reality check and SPA test have no closed form; they resample the data to build the distribution of the best rule's performance under the null.

Bonferroni is conservative when tests are correlated, as backtest variants on shared data always are; bootstrap approaches like White's reality check handle the correlation structure implicitly by resampling the actual return series.

Some practitioners instead control the false discovery rate (e.g. Benjamini-Hochberg), tolerating a known fraction of false positives rather than guarding against any.

How traders use it

• As a research-program discipline: log every configuration tested, estimate the effective trial count, and require the surviving strategy to clear the corrected threshold rather than the naive one.
• Via bootstrap tests when evaluating a family of rules against a benchmark: White's reality check or the SPA test answers whether the best rule's outperformance exceeds what the luckiest member of that family would show under the null.
• Through the deflated Sharpe ratio, which packages a multiple-testing correction specifically for Sharpe ratios and is often the most convenient form for strategy selection.
• As a skeptical lens on published and vendor results: asking how large the unreported search was, and what threshold the result would need to clear after correction, deflates many impressive-looking findings.
• With awareness of the trade-off: overly harsh corrections reject genuine but modest edges, which is one reason practitioners pair them with independent evidence such as forward confirmation rather than relying on any single gate.

Multiple-testing correction vs related concepts