https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2326253

Abstract

Most firms and portfolio managers rely on backtests (or historical simulations of performance) to select investment strategies and allocate them capital. Standard statistical techniques designed to prevent regression over-fitting, such as hold-out, tend to be unreliable and inaccurate in the context of investment backtests. We propose a framework that estimates the probability of backtest over-fitting (PBO) specifically in the context of investment simulations, through a numerical method that we call combinatorially symmetric cross-validation (CSCV). We show that CSCV produces accurate estimates of the probability that a particular backtest is over-fit.