Curve Fitting and Adaptation
Previous  Top  Next

Curve fitting, also known as over-optimizing, is the pariah of strategy optimization. Curve fitting is the process of optimizing a strategy to such a degree that you get good back testing results only because you have tailored them for a specific set of price points, and not because you have unearthed some truism about the price series that will remain true in the future. You will know that you have curve-fitted when, as you test future data, your strategy yields drastically different results, usually for the worse.

Many so-called experts will tell you than you should first have a sound basis for your strategy and then use parameters that work. But if you are not an expert and want to create a strategy, how can you do it and become an expert? Naturally you will try various things to see what works, and learn from that. And you will do it in the hopes that you find something that the experts have not, and in the process learn what works and what doesn't work. The major problems that people encounter in these endeavors are:

·not enough processor power to try all of the combinations of things they want to try  
·an inability to determine if the results obtained are curve-fitted  

Optimax addresses both of these points. We already know that Optimax's genetic search algorithms reduce optimization time to a minimum. To address the second point, Optimax uses a new technique: price modulation. Most people use a technique called walk-forward testing to address this issue. Walk-forward testing has both pros and cons, and we also plan to add this capability to Optimax in the future, so it is worthwhile to discuss here and compare it to our solution of price modulation.

Pros and Cons of Walk-Forward Testing
To overcome the problem of curve fitting, many people use walk-forward testing. This is a process by which you optimize over a historical chart period, holding in reserve a segment of the chart data, usually the most recent data. After you have optimized, you test again using the reserved data. If the strategy performs well on the data that wasn't used during optimization, then the strategy is probably not curve fitted. That's great, as long as your test goes well.

But what if your walk-forward test fails on the unseen data?

The biggest problem now is not what you have done so far, but what you will do next. If your test fails, do you throw all your work away and start from scratch? Or do you work with your strategy some more and see if you can come up with another approach that yields satisfactory results on the unused segment of data? If you are even only half-way human, you will probably modify your strategy and try again. Unfortunately, at that point, you are no longer able to say that your walk-forward test is reliable, because you are now trying to change your strategy so that the unused segment of data yields acceptable results as well. Once your strategy has seen the unused data, it is no longer unused. If you continue testing you are now curve-fitting, no matter how much thou dost protest.

So, the only reliable way to use a walk-forward test is as a final go/no-go test. If it fails, you should reject the basis of the strategy and try again from scratch. Else, you have fallen into the curve-fitting trap - albeit via a circuitous route.

As if that weren't enough , there is yet another problem with walk-forward testing - it requires you to optimize over less data than you currently have. An inefficiency arises because you've performed your optimization with some of the data missing - you haven't used it all and thus you couldn't have optimized your strategy to take full advantage of all of the price action.

So what's a poor strategy-designer to do?

That's where price modulation, available only in Optimax, comes in. Using the price modulation options on the Optimization Settings window, you can cause the price series to change during the optimization process, causing each individual to see a slightly different price series. You do this by applying a modulation wave and a random noise function at the same time, and this will cause the price series to be different for every test during the optimization. The strength of the modulation you apply determines the amount of difference, thus reducing the possibility of curve fitting by the amount of variability you inject into the price series. Injecting the maximum amount of variability will reduce the likelihood of curve fitting nearly to zero.

The reason we chose to implement this technique before walk testing is because there are more advantages in using modulation than walk testing:

·you can use all of your data for testing  
·you can adjust the amount of variability, and thus "stress-test" your strategy  
·you can add a random walk to the price series, thus performing "what-if" tests with your strategy  

There is always a danger in optimizing when you are optimizing over a static data stream. With random walk price modulation, that series is no longer static, and you now have a way of testing the adaptability of your strategy as well as the profitability. Since the stock market isn't static, but is dynamic, optimizing over a static datastream is an unrealistic way to back test. Testing over a dynamic datastream increases the reality of the tests and adds a never-before-available level of integrity of your simulations.

It is extremely important to modulate the price series during the genetic optimization.