Backtesting Limitations (Manual and Automated)

Backtesting is the best way to verify that a trading strategy has an edge and optimize that strategy to meet your goals.

Despite its importance, backtesting does have its limitations.

The primary constraints of manual backtesting are that it requires discretionary input from the trader and is not highly scalable. Automated backtesting is limited by the quality of the code and it lacks flexibility. 

I’ve been backtesting since 2008 and it has helped me tremendously and has helped countless traders that I’ve met over the years.

My goal in this article is to help you understand the details of the limitations of the backtesting method you use, so you can minimize the negative impact they can have on your results.

These limitations apply to backtesting trading strategies in any market.

Alright, let’s get into it…

Limitations of Manual Backtesting

Manual backtesting involves a trader simulating a trading strategy on historical data by manually checking whether each trade would have been a success or a failure according to the strategy’s criteria.

This process requires the trader to scroll through past market data, apply the trading strategy rules, and record the outcomes of each hypothetical trade.

There are many software solutions that can dramatically speed up this process, or a trader can choose to simply record their results on a spreadsheet.

Here are the limitations of manual backtesting that you need to be aware of.

Human Discretionary Bias

Arguably, the biggest limitation of manual backtesting is that human bias is introduced into the backtesting process.

This is not necessarily a bad thing however.

The benefit to manual backtesting is that it can test discretionary trading strategies, of which there are many more than automated strategies.

So if you backtest manually, the key to success is applying your logic consistently.

Even though you are making discretionary decisions, do your best to understand what a good trade looks like.

But even the most diligent trader will have some variability in their backtesting, so it’s important to test the same strategy several times before trading it live.

It’s also worth noting that since there is individual discretion in manual backtesting, results can vary greatly between traders.

Therefore, it’s always best to test a strategy for yourself and not rely soley on the results of others.

Relatively Slow Process

A big downside of manual backtesting is that it’s inherently time-consuming.

You must painstakingly go through historical price data, applying a strategy’s criteria to determine entry and exit points.

This process is not only slow but also prone to errors, as it relies heavily on your attention to detail and patience.

So when backtesting manually, it’s important to focus on testing one strategy at a time.

The upside to this slower process is that you get a more detailed look at each trade and that can help you develop new optimizations faster than with automated backtesting.

Not Scalable

Since manual backtesting is slow, it cannot be scaled efficiently.

As traders seek to validate their strategies across different instruments, timeframes, and market conditions, the manual process becomes exponentially more cumbersome.

So if you’re going to backtest manually, you have to be resigned to the fact that you’ll only be able to test a few markets and timeframes at a time.

But this can be a blessing in disguise because it will allow you to become an expert in a few core markets, instead of a generalist in many markets.

Risk of Overfitting

The second major limitation is the risk of overfitting.

Overfitting occurs when a strategy is too closely tailored to past data, making it perform exceptionally well on historical data but poorly in real-world trading.

Manual backtesting, with its reliance on subjective judgment, increases the risk of introducing biases into the testing process.

You may unconsciously select data that confirms the strategy’s effectiveness or overlook data that contradicts it.

This selection bias can lead to over-optimistic results that do not accurately reflect the strategy’s real-world performance.

Therefore, it’s important to test your strategies on as much historical data as possible and not “cherry pick” the best performing periods.

Potential Miscalculations

Real-world trading involves factors such as slippage, transaction costs, and varying liquidity, which can significantly impact the profitability of a strategy.

Manual backtesting can oversimplify these aspects and make strategy look much more profitable than it really is.

Without a realistic representation of market conditions, the results of manual backtesting can be misleading, painting an inaccurate picture of a strategy’s potential success.

To overcome this, always check that you’re using realistic settings for things like commission, spread and slippage.

If these things are hard to factor in, you can always manually lower the return of your strategies slightly to account for unknown variables.

Limited Complexity

Another critical limitation of manual backtesting is the difficulty in testing complex or quantitative strategies.

If you’re going to backtest discretionary trading strategies, you have to stick to methods that are simple and easy for you to calculate and execute.

Manual backtesting is not suited to handle high-frequency trading strategies or those that rely on complex analysis.

Although it may seem appealing to execute super complex, uber clever strategies, in my experience it’s often the simple strategies that do best.

Limitations of Automated Backtesting

Automated or programmatic backtesting has revolutionized the way traders develop and evaluate their strategies, leveraging historical data to predict future performance without manual intervention.

By simulating trades based on specific criteria and algorithms, this method offers efficiency, precision, and the ability to test complex strategies across multiple datasets and timeframes.

Despite these advantages, automated backtesting is not without its limitations, which can impact the reliability of the results and the ultimate success of trading strategies.

Overfitting Risk

One of the primary challenges with automated backtesting is the risk of overfitting.

Overfitting occurs when a model is excessively complex, with numerous rules or parameters that are tailored to perform exceptionally well on historical data but fail to perform in the future.

This phenomenon leads to inflated backtest performance that cannot be replicated in live trading.

An example of this is the story of LTCM (Long-Term Capital Management), a hedge fund that relied heavily on complex mathematical models.

Despite the brilliance of its team, which included Nobel laureates, LTCM folded in 2000, due to a combination of excessive use of leverage and strategies that were overfit to past market conditions.

Look Ahead Bias

Another significant limitation is the “look-ahead” bias, which occurs when a strategy inadvertently utilizes future information in its trading decisions, leading to unrealistic backtest results.

This can happen through programming errors or when the data set includes future data not available at the time of trade execution.

The illusion of extraordinary returns generated by such biases can be misleading and result in substantial losses when the strategy is applied in real-time trading.

Trading Fees Miscalculation

Trading fees also present a hurdle for automated backtesting.

Historical data may not fully capture the market’s liquidity, bid-ask spreads, and slippage that can significantly affect transaction costs and execution.

Automated backtests often assume ideal trading conditions, neglecting these real-world trading expenses.

This oversight can lead to an underestimation of costs and an overestimation of strategy performance.

Lack of Human Input and Intuition

Furthermore, the absence of human intuition and experience is a notable drawback of automated backtesting.

While automated systems excel in processing vast amounts of data and executing predefined strategies, they lack the ability to interpret nuanced market signals or adjust to unforeseen events.

In other words, they are only able to run the instructions they are given.

Famous traders like Jim Simons, the founder of Renaissance Technologies, have successfully combined automated trading with human oversight.

Simons, a mathematician, and his team developed sophisticated algorithms that have consistently outperformed the market.

However, the success of Renaissance Technologies also relies on continuous refinement of its models and the expert judgement of its team, highlighting the importance of blending automated strategies with human insight.

Technology Risk

The dependence on technological infrastructure can pose serious risks.

Once an automated strategy is backtested, live trading requires robust computing resources, stable internet connectivity and continuous human oversight.

Since computer programs only follow instructions they are given, if certain scenarios are not accounted for, any failure in these systems can lead to significant losses, especially with high-frequency trading strategies.

One example is what happened to Knight Capital Group in 2012.

A glitch in their computer programs caused them to amass huge losses and almost sent them into bankruptcy.

Final Thoughts on Backtesting Limitations

Even though there are limitations to both manual and automated backtesting, there’s no doubt that some sort of backtesting has to be done to verify and optimize trading strategies.

Risking real money on an untested strategy is like hiking in the mountains without a map.

You don’t know a proven path, you’re very likely to get lost on your hike and maybe not even make it back.

Therefore, the solution is to pick the backtesting method that best suits your skills and goals.

Then understand the limitations of your chosen method and minimize the negative impact of its limitations.

If you want to learn the tools and techniques for backtesting both manual and automated trading strategies, continue reading my tutorial on how to backtest a trading strategy.