Avoiding Common Problems with Backtesting

Backtesting options is a notoriously difficult process, prone to user error, overfitting, and bad data. That’s why we’ve done all the hard work for you, and ran millions of backtests spanning all types of strategies. After 20 years of being in the backtesting business, we’ve learned a thing or two about common problems traders run into. That’s why we’ve paid close attention to the following problems: a. Overfitting and b. Quality of data and returns. The remainder of this blog covers our approach to solving those two problems.

Find Similar Backtests

At ORATS, we believe in the value of learning from the past to better navigate the future. That's why we've incorporated the feature to find similar backtests in our tool. This allows you to identify and compare your current backtest with other backtests that have similar parameters. It provides you with an immediate context and helps identify potential issues that might not be obvious in isolation. It's a proven way to avoid overfitting, as it encourages the use of parameters that have worked well across different market conditions, rather than those that only performed well in a single backtest. In the example below, I’m looking for similar backtests with the same DTE, strike deltas, spread / stock, and spread / strike diff, but with different entry and exit triggers. This way, I can check for overfitting by seeing if those similar backtests had a wildly different outcome.

Path Dependency

Path dependency refers to how the order and timing of trades can significantly affect a strategy's performance. Ignoring this aspect can lead to misleading results and unrealistic expectations. Our backtester tool takes into account path dependency in a unique way, by only putting on one trade per day, every day it meets the entry criteria. This means you can have 10 trades on at the same time, all entered one day after the previous one. While this isn’t how most people trade in real life, it provides more accurate performance metrics because it eliminates any statistical bias that would occur if you only entered one trade at a time. For example, starting a backtest on the first day of the year might have very different results than starting the backtest a few weeks later.

Entry Triggers

Ensuring you have accurate entry triggers is another critical aspect of successful backtesting. Inappropriate triggers can result in substantial discrepancies between the backtest results and actual performance. We’ve identified five entry triggers that represent a variety of different market conditions. These are:

• VIX Price

• Simple Moving Average

• 14d RSI

• IV Percentile 1 Year

• Slope Percentile 1 Year

In addition, we limit the trigger choices to just a few preset values for each one - low, moderate, and high. If more choices were tested, we run the risk of overfitting based on very niche market conditions. We also limit each test to two entry triggers, so that we don’t have so many that the backtest rarely runs.

Performance Metrics

To ensure that you're not falling into the trap of overfitting or basing your strategies on poor quality data, our backtester tool provides a comprehensive suite of performance metrics. It delivers detailed reports about the performance of your strategies, such as the Sharpe ratio, drawdowns, and various other risk-adjusted return metrics. Of these metrics, “Percent of Time in Market” and “Total Strategy Trades” can be used to filter down backtest results to those that have a large sample size. For example, if you set the minimum Percent of Time in Market to 40%, this will limit results to backtests that were in the market at least 40% of the time. Doing this ensures that the backtests you find have a statistically sound number of trades.

Conclusion

At ORATS, we understand that successful backtesting is both an art and a science. With our new backtester, we've combined decades of market expertise with advanced technological solutions to help you navigate the common problems associated with backtesting. By addressing issues such as overfitting, data quality, path dependency, and the accuracy of entry triggers, we've developed a tool that's capable of delivering reliable and realistic backtest results. Step into the future of backtesting with ORATS, and let your trading strategies be guided by data, not guesswork.