Randomization Tests

What are randomization tests?

Randomization tests judge a backtest not against zero but against luck. The idea is to inject controlled randomness into some component of the strategy, its trade sequence, its entry timing, or the price data itself, re-run the test many times, and build a distribution of results that pure chance could have produced. The actual backtest is then read as one point against that distribution: if a meaningful fraction of the randomized runs match or beat it, the observed edge is indistinguishable from noise. Traders know the main variants as Monte Carlo trade reshuffling and the random-entry baseline.

The method exists because a single backtest is a single draw from history, and optimizers are efficient at finding settings that fit that one draw. Randomization borrows the logic of permutation testing from statistics: rather than assume a distribution for what luck looks like, it manufactures one empirically. This makes it a natural companion to holdout methods, since it attacks a different failure mode. A holdout asks whether the edge persists on unseen data; a randomization test asks whether the edge was ever more than chance on the seen data.

The variants probe different things. Reshuffling the order of closed trades keeps every trade result but scrambles their sequence, which reveals how much of the equity curve's smoothness and drawdown profile was path luck; it is the standard way to turn a single maximum drawdown into a distribution of plausible ones. Random-entry baselines keep the exit and position-sizing rules but replace entries with coin flips, testing whether the entry logic contains information or whether the exits and risk management were doing the work. Permuting or regenerating the price series itself, as in Monte Carlo price paths, tests the full system against data where any exploitable structure has been destroyed.

How it's calculated

Randomization tests share one canonical output, the empirical p-value:

Many implementations use (count + 1) / (N + 1) so that p is never exactly zero.

For loss-type metrics such as drawdown, the inequality is reversed.

How traders use it

• Trade reshuffling converts one backtest drawdown into a distribution of drawdowns, and position sizing is then set against a high percentile of that distribution rather than the single historical figure, feeding directly into drawdown statistics.
• Random-entry runs act as a null model for entry signals: if the strategy barely beats coin-flip entries with the same exits, the entry logic is contributing little and the system is really an exit and risk framework.
• Data-permutation runs vet the whole pipeline for overfitting: a system that still profits on scrambled prices has fit an artifact of the test, not the market.
• As a screening gate in strategy research, where candidates whose results fall inside the randomized distribution are discarded before any capital or further tuning is spent.
• The tests have limits worth respecting: reshuffling assumes trades are independent, which understates risk when returns are autocorrelated, and no randomization scheme can detect look-ahead bugs it was not designed to break.

Randomization tests vs adjacent methods