Evaluation in a Nutshell

Previous  Top  Next

Hindsight is always 20/20. Creating a strategy that gives good results from historical data is only the start of your task - your strategy must continue to perform in the future with data it has never before seen in order for you to make money. If the stock market changes - and it will - this will affect your results, usually for the worse. To avoid nasty surprises in the future, here is a checklist of things to look for while optimizing.

Evaluation Checklist

Goal What to look for stable inputs A small change in the inputs should not result in a large change in the equity. consistent returns The equity curve should resemble a straight line rising at a 45-degree angle. convergence, not divergence The curves should become more similar as time passes. even search space coverage The dots and lines on the ISpace and SSpace indicators should be fairly evenly spaced with some clustering. few bands The more bands within the ISpace indicator, the more likely it is that you are curve fitting. consistent results from multiple optimization runs If the results are different each time you run the optimization, it is likely you are not covering the entire search space in a single run.

Stable Inputs
Look for green bands in the ISpace indicators which indicate areas of input stability. The wider the band, the more stable is the area of values and the smoother the change will be from a winning to a loosing strategy. Remember, if you change the inputs a little, the equity curve should only change a little. If it changes a lot, this indicates that you are probably curve-fitting. The more drastic the change, the more likely it is that your strategy will fail in the future when it encounters new market conditions. Look for the green bands to avoid this pitfall.

Consistent Returns
Look at the equity curves - if they look like a straight line drawn at a 45-degree angle then the returns are consistent. The absolute net profit means little without consistency. Using OMX_F_MSRatio or a similar measure of consistency for your fitness will increase your chances of finding consistent returns. Remember, if your strategy's returns are consistent from period to period, then it is much more likely to continue to perform consistently in the future. If the results vary widely from period to period, then it is likely to vary even more widely in the future. Look for straight-line equity curves to side-step this common error.

Convergence, Not Divergence
Look for the similarity in equity curves as the optimization progresses, indicating the optimization is coming to a natural end. These curves should become more and more like each other. This is called convergence. If they stay widely diverse or become more diverse over time, then the settings are not appropriate for your strategy and you have not successfully used Optimax to assess your strategy.

Even Search Space Coverage
Look for a uniform coverage of the SSpace indicator, indicating that the search space has been well-covered and no major input combinations remain unsearched. Big gaps of white indicate that some part of the terrain was not covered. Coverage will never be perfectly uniform - there will always be some clustering of purple - because Optimax concentrates it's search in the higher fitness areas.

Few Bands
Look for few bands within the ISpace indicator panels. The fewer the bands, the less probable it is that you are curve-fitting. Many bands indicate a complex fitness terrain and a higher probability of curve-fitting and thus future failure of the strategies.

Consistent Results
Run the same optimization multiple times and look for consistent results. If the banding occurs in the same places at the end of each optimization, then you know that Optimax has adequately covered the entire search space each time. But if the results are different each time, then the space was not adequately searched. To correct this, increase the Initial Population Size setting, or decrease one of these parameters: Genetic Complement number, Mutation Rate, Elite Percent Taper, Maximum Life Span, Tournament Group Size, Roulette Scaling Factor, or increase Top Percent.