Multiple-testing Correction

Multiple-testing Correction puts the search itself under examination. The build backtests a fully logged family of rule variants — N lookbacks of one transparent rule — so the trial count every multiple-testing correction formula needs is known exactly, then shows what the family's best result is worth once the search is priced in. The pane draws each trial's relative equity curve inside a shaded envelope, a bootstrap null distribution of the luckiest rule beside the last bar, and a verdict dashboard.

How to Trade the Multiple-testing Correction?

• Verdict line: the bottom row states the conclusion, from naive-only pass — consistent with data mining — to clearing both the corrected bar and the reality check; transitions fire alerts.
• Best rule vs the null: the observed best is drawn against the bootstrap distribution of the luckiest recentred rule; below the dashed corrected bar, luck alone accounts for it.
• Effective N row: variants scored on shared data are correlated, so the estimated number of independent trials — not the raw count — drives the default correction.

Multiple-testing Correction Settings

• Rule Family (default Trend (Close vs SMA)), Rules Tested (N) (default 15) and Lookback Range (defaults 10 to 150): the logged search.
• Positioning (default Long / Short) and Benchmark (default Cash (Zero Return)): trial scoring.
• Family-wise Alpha % (default 5) and Analysis Window (Bars) (default 500): false-positive budget and shared sample.
• Primary Correction (default Sidak (Effective N)): the threshold behind the verdict and alerts.
• Bootstrap Replications (default 200), Expected Block Length (default 10) and Random Seed (default 1): the stationary bootstrap.
• Distribution Bins (default 15) plus table and style toggles.

Frequently Asked Questions

How is this different from walk-forward analysis?

Walk-forward analysis attacks selection bias with data the optimizer never saw; a correction scales the required evidence with how many things were tried on the same data. Complementary questions — serious validation uses both.

Why do the Effective N variants exist?

Bonferroni and Sidak treat trials as distinct, but variants backtested on one shared history are heavily correlated, so the raw count over-penalizes. The Effective N forms divide alpha by the estimated number of independent trials instead, recovered from average pairwise correlation.

Are the bootstrap p-values reproducible?

Yes — the generator is seeded: the same Random Seed redraws the identical distribution on every load, and changing it checks a conclusion against a fresh resampling.