Bias Taxonomy

What is a bias taxonomy?

A bias taxonomy is a structured checklist of the systematic errors that inflate backtested performance: the recurring ways a historical test can flatter a strategy that has no real edge. The best-known entries are look-ahead bias, where the simulation uses information that was not available at decision time; survivorship bias, where the tested universe silently excludes delisted or failed instruments; data snooping, where an edge is manufactured by testing many variations against one history and keeping the winner; and repainting, where an indicator revises its past values so historical signals look cleaner than anything a live trader ever saw.

The taxonomy exists because backtests rarely fail from one dramatic mistake. They fail from an accumulation of small, structural flaws, each of which biases results in the same direction: upward. Unlike random noise, these errors do not cancel out, so a backtest containing several of them can show a smooth equity curve for a strategy that loses money live. Naming the failure modes turns a vague worry about curve-fitting into an auditable checklist.

Traders care because each bias has a distinct fingerprint and a distinct cure. Look-ahead bias is fixed by strict point-in-time data and careful signal timing; survivorship bias by using delisting-inclusive datasets; repainting by repaint-safe engineering that only reads confirmed bars; and data snooping by a multiple-testing correction or an honest in-sample / out-of-sample split. Diagnosing which bias is present matters more than knowing that some bias exists.

How traders use it

• As a pre-mortem checklist before trusting any backtest: walk through look-ahead, survivorship, snooping, repainting, and cost omissions one by one, and document how each was ruled out or bounded.
• As a debugging guide when live results undershoot the backtest: the gap's character often points to the culprit, for example an immediate collapse suggests repainting or look-ahead, while slow erosion suggests snooping or optimistic costs.
• As a review standard when evaluating someone else's published results, since third-party backtests rarely disclose universe construction, signal timing, or how many variants were tried before the reported one.
• As a design constraint from the start: choosing point-in-time data, confirmed-bar signals, and a limited pre-registered set of parameter trials prevents most biases more cheaply than detecting them afterward.
• Alongside quantitative audits such as the probability of backtest overfitting, which estimate how much damage the snooping entry of the taxonomy has already done.

Bias taxonomy vs related concepts