3 Simple Ways To Reduce The Risk Of Curve-fitting

Curve-fitting is almost certain death for a trading or investment strategy.

So, what is curve-fitting?

Well, you know when you test a trading or investment hypothesis, fall in love with the historical results, and then the idea fails to generate similar (or even positive) returns once you decide to trade it live?

Most of the pitfalls of system trading, and trading in general, can be avoided or mitigated following these three simple techniques or rules of thumb.

  1. Use out of sample data! Out of sample data is simply withholding some of the data in your “test” period for further evaluation. For example, you have ten years of historical data and opt to put the last 30% in your back pocket. You develop a great trading strategy on the first seven years of the data set and then whip out your “out of sample” data (remaining 30% from your back pocket) and validate your findings. If the strategy fails to produce similar results in the out of sample data then you can be almost certain you have curve-fit to the first seven years of your data set. Below is chart of a strategy built using Build Alpha that highlights the out of sample period. You like to see similar growth (and characteristics) in both the in-sample and out-of-sample data.

oos_highlights2. Make sure your strategy has enough occurrences or trades. This can be simply explained with a coin flip example. If you flip a coin 10 times and it lands on heads 7 times you cannot be certain you do or do not have a rigged coin. However, if you flip a coin 10,000 times and it lands on heads 7,000 times you can have very high confidence it is indeed a rigged coin. In trading, if a strategy has 30 trades then it is unlikely you would have high confidence that what you have found is legitimate. However, if a strategy holds up over 1,000 or 3,000 plus trades then you can have higher confidence you’ve discovered true edge.

Below is a photo of only 30 coin flips and below that is a photo of six different trials of 100,000 coin flips. You can see after a large number of occurrences things tend to converge toward the true expectation (also known as the Law of Large Numbers).

smallsamples_30_3
smallsamples_100000_13. Validate your strategy across other markets. If a strategy works on only one market it has a higher chance of being curve-fit to the data set than if a strategy performs profitably on a handful of markets. I am not saying that a strategy that only works on one market is curve-fit – as there are many nuances, different players, and idiosyncrasies that exist within each market. However, if a strategy performs across markets then you can certainly have higher confidence that it is not curve-fit.

acrossmarkets

Thanks for reading,

Dave

Thanks for reading,
Dave


Leave a Reply

Your email address will not be published. Required fields are marked *

Login


Username
Password
(close)

Create an Account!


Username
Email
Password
Confirm Password
Want to Login? (close)

forgot password?


Username or Email
(close)

Risk Disclosure

FUTURES AND FOREX TRADING CONTAINS SUBSTANTIAL RISK AND IS NOT FOR EVERY INVESTOR. AN INVESTOR COULD POTENTIALLY LOSE ALL OR MORE THAN THE INITIAL INVESTMENT. RISK CAPITAL IS MONEY THAT CAN BE LOST WITHOUT JEOPARDIZING ONES FINANCIAL SECURITY OR LIFE STYLE. ONLY RISK CAPITAL SHOULD BE USED FOR TRADING AND ONLY THOSE WITH SUFFICIENT RISK CAPITAL SHOULD CONSIDER TRADING. PAST PERFORMANCE IS NOT NECESSARILY INDICATIVE OF FUTURE RESULTS.

Hypothetical Performance Disclaimer

HYPOTHETICAL PERFORMANCE RESULTS HAVE MANY INHERENT LIMITATIONS, SOME OF WHICH ARE DESCRIBED BELOW. NO REPRESENTATION IS BEING MADE THAT ANY ACCOUNT WILL OR IS LIKELY TO ACHIEVE PROFITS OR LOSSES SIMILAR TO THOSE SHOWN; IN FACT, THERE ARE FREQUENTLY SHARP DIFFERENCES BETWEEN HYPOTHETICAL PERFORMANCE RESULTS AND THE ACTUAL RESULTS SUBSEQUENTLY ACHIEVED BY ANY PARTICULAR TRADING PROGRAM. ONE OF THE LIMITATIONS OF HYPOTHETICAL PERFORMANCE RESULTS IS THAT THEY ARE GENERALLY PREPARED WITH THE BENEFIT OF HINDSIGHT. IN ADDITION, HYPOTHETICAL TRADING DOES NOT INVOLVE FINANCIAL RISK, AND NO HYPOTHETICAL TRADING RECORD CAN COMPLETELY ACCOUNT FOR THE IMPACT OF FINANCIAL RISK OF ACTUAL TRADING. FOR EXAMPLE, THE ABILITY TO WITHSTAND LOSSES OR TO ADHERE TO A PARTICULAR TRADING PROGRAM IN SPITE OF TRADING LOSSES ARE MATERIAL POINTS WHICH CAN ALSO ADVERSELY AFFECT ACTUAL TRADING RESULTS. THERE ARE NUMEROUS OTHER FACTORS RELATED TO THE MARKETS IN GENERAL OR TO THE IMPLEMENTATION OF ANY SPECIFIC TRADING PROGRAM WHICH CANNOT BE FULLY ACCOUNTED FOR IN THE PREPARATION OF HYPOTHETICAL PERFORMANCE RESULTS AND ALL WHICH CAN ADVERSELY AFFECT TRADING RESULTS.