Probability of Backtest Overfitting

What is the probability of backtest overfitting?

The probability of backtest overfitting (PBO) is a diagnostic that estimates how likely it is that a strategy selected for its backtest performance is actually a poor performer out-of-sample. Introduced by Bailey, Borwein, Lopez de Prado, and Zhu, it reframes overfitting as a question about selection: given the full set of configurations a researcher tried, how often does the one that looks best in-sample fail to stay above median when scored on data it was not selected on?

The estimate comes from a procedure called combinatorially symmetric cross-validation (CSCV). The performance history of all N trials is cut into S equal blocks, and every way of assigning half the blocks to a training role and half to a testing role is enumerated. For each split, the trial that ranks best on the training half is identified, and its rank on the testing half is recorded. If selection were finding real edge, the in-sample winner should tend to rank well out-of-sample too; if it were merely harvesting noise, its out-of-sample rank should scatter like any other trial's. PBO is the fraction of splits in which the in-sample winner lands in the bottom half out-of-sample.

Traders care because PBO evaluates the research process rather than a single strategy, which is where overfitting actually lives. A high PBO says the selection procedure itself is unreliable: whatever configuration it crowns is likely a fluke, however good its equity curve. It complements threshold-based tools like the deflated Sharpe ratio by requiring no distributional assumptions, at the cost of requiring the full trial-by-trial performance record.

How it's calculated

PBO is estimated by CSCV over the matrix of period returns for all N trials, using a chosen performance metric (commonly the Sharpe ratio).

Blocks are kept contiguous in time to respect serial dependence; the symmetric design means every observation serves in both roles across combinations.

The same framework yields related diagnostics, such as the slope of out-of-sample versus in-sample performance across combinations, which measures performance degradation.

How traders use it

• As an audit of a parameter sweep: retain the per-period returns of every configuration tested, run CSCV, and treat a PBO well above a chosen tolerance (many practitioners become uncomfortable beyond roughly 0.2 to 0.5) as evidence the sweep is selecting noise.
• As a comparison between research processes: two teams can produce similar-looking best backtests while their PBOs reveal that one selection procedure generalizes and the other does not.
• As a companion to the deflated Sharpe ratio: PBO checks rank stability without distributional assumptions, the DSR checks magnitude against a luck-based hurdle, and passing both is materially stronger evidence than passing either.
• With honest inputs: PBO only sees the trials it is given, so omitting failed experiments from the matrix biases it downward, exactly the manipulation it was designed to expose.
• Within its limits: CSCV's block reshuffling weakens strict temporal ordering, so PBO complements rather than replaces walk-forward analysis and genuine forward confirmation.

Probability of backtest overfitting vs related concepts