Backtesting—the evaluation of an investment strategy against historical data to assess its past performance—is one of the most widely used and most systematically misinterpreted tools in investment analysis. A strategy that performs well in backtesting is not demonstrated to be a good strategy; it is demonstrated to be a strategy that would have performed well in the specific historical period tested, which is a substantially weaker claim whose relationship to future performance depends on assumptions that the backtest itself cannot validate.
The fundamental problem with backtesting is the data mining bias, also called overfitting. Any historical dataset contains patterns—some of which reflect genuine persistent regularities, and many of which are artefacts of the specific historical period, statistical noise, or combinations of both. A sufficiently flexible strategy-testing framework, applied to a sufficiently rich dataset, will identify strategies that perform extremely well in the historical period simply by happening to match the noise patterns in the data. These strategies will not perform well going forward, because the noise patterns they matched will not recur—but the backtest cannot distinguish between strategies that match genuine regularities and strategies that match historical noise.
The emotional impact of a strong backtest is significant and largely independent of its actual evidentiary value. A backtest showing that a strategy would have returned fifteen percent annually over the past twenty years, with a Sharpe ratio well above one and a maximum drawdown of only twelve percent, produces a compelling sense of confidence that the strategy is sound—even if every financial economist who examined it would identify it as a data-mined artefact. The visual presentation of the historical returns, the apparently objective nature of the quantitative analysis, and the reassurance of seeing a historical record that would have generated genuine wealth combine to create confidence that the evidence does not support.
The specific ways in which backtests systematically overstate expected forward performance are well documented. They typically do not account for the transaction costs that implementing the strategy would have generated. They do not account for the market impact of the strategy at the scale of implementation, which would have moved prices against the strategy in any period of significant deployment. They are typically constructed with the benefit of hindsight about which parameters would have worked best—a form of implicit overfitting that produces better historical results than any reasonable forward-looking strategy selection process would have achieved.
The appropriate use of backtesting is as a falsification tool rather than a confirmation tool. A backtest that shows a strategy would have failed historically is strong evidence that the strategy has a problem; a backtest that shows a strategy would have succeeded historically is weak evidence that the strategy will succeed going forward. The investor who uses backtests to eliminate strategies with clear historical problems—strategies that fail even in the most favourable historical environments—is using the tool appropriately. The investor who uses backtests to build confidence in a strategy's future prospects is using a tool that provides less evidentiary support for that confidence than it appears to.