Noise Test

First, let’s talk about overfitting and what it is. I will simply define it as fitting a function or model to data so well that the function or model is not generalizable and cannot/will not work on another data set.  In trading terms, it is designing a strategy that trades historical data so well that it will surely fail on new data.

The first and most obvious test to prevent curve fitting is out of sample testing. In other posts I’ve described this, but in short, it is simply withholding a portion of your data to “validate” the strategy. For example, you have 10 years worth of data and you designate the first or last 30% to be out of sample. You build your model on the remaining seven years of data and then validate it on the withheld 30% or out of sample data.

However, and for various reasons, this might not be enough to prevent against curve fitting. There are two parts to the underlying data: the signal and the noise. It is often said that when something is curve fit it is “fitted to the noise” and does not capture the underlying signal. The Noise Test is a test to see how close we have fit to the “noise” in our price data.

The Noise Test adjusts the original price data by differing amounts (user selected) and creates hundreds of new price series. For example, the trader can select to change up to 40% of the opens, 60% of the highs, 50% of the lows, and 75% of the closes. Additionally, the trader can select by how much he’d like to adjust the price data – let’s say by up to 33% of the average true range. Here’s what this exact configuration would look like in Build Alpha.

noiseconfigs

Below I’ve attached photos that show 4 “noise altered” price series created with the settings above for the S&P500. The charts below display the period of the August 2015 ETF crash. You can see how some bars have changed and some have even turned positive when originally negative and visa versa.

It still mimics the original S&P500 data and that is ideal as we want to keep certain elements of the original data such as the volatility clustering, e.g. However, we’ve change the amounts of noise in each newly created series by adding or subtracting various amounts.

*Actual Price Data*
*Actual Price Data*
Noise Altered
Noise Altered
Noise Altered
Noise Altered
Noise Altered
Noise Altered
Noise Altered
Noise Altered

The Noise Test will then trade the strategy on each newly created price series and display the resulting equity curves. If the original strategy was “fitted to the noise” then the strategy should not maintain profitability as we adjust the noise in the data. However, if the Noise Test results remain profitable then we can have increased confidence a strategy was trading the underlying signal and not the “noise”.  Below is the output of the Noise Test in Build Alpha.

noisetest1

Thanks for reading,
Dave


2 thoughts on “Noise Test

Leave a Reply

Your email address will not be published. Required fields are marked *

Login


Username
Password
(close)

Create an Account!


Username
Email
Password
Confirm Password
Want to Login? (close)

forgot password?


Username or Email
(close)

Risk Disclosure

FUTURES AND FOREX TRADING CONTAINS SUBSTANTIAL RISK AND IS NOT FOR EVERY INVESTOR. AN INVESTOR COULD POTENTIALLY LOSE ALL OR MORE THAN THE INITIAL INVESTMENT. RISK CAPITAL IS MONEY THAT CAN BE LOST WITHOUT JEOPARDIZING ONES FINANCIAL SECURITY OR LIFE STYLE. ONLY RISK CAPITAL SHOULD BE USED FOR TRADING AND ONLY THOSE WITH SUFFICIENT RISK CAPITAL SHOULD CONSIDER TRADING. PAST PERFORMANCE IS NOT NECESSARILY INDICATIVE OF FUTURE RESULTS.

Hypothetical Performance Disclaimer

HYPOTHETICAL PERFORMANCE RESULTS HAVE MANY INHERENT LIMITATIONS, SOME OF WHICH ARE DESCRIBED BELOW. NO REPRESENTATION IS BEING MADE THAT ANY ACCOUNT WILL OR IS LIKELY TO ACHIEVE PROFITS OR LOSSES SIMILAR TO THOSE SHOWN; IN FACT, THERE ARE FREQUENTLY SHARP DIFFERENCES BETWEEN HYPOTHETICAL PERFORMANCE RESULTS AND THE ACTUAL RESULTS SUBSEQUENTLY ACHIEVED BY ANY PARTICULAR TRADING PROGRAM. ONE OF THE LIMITATIONS OF HYPOTHETICAL PERFORMANCE RESULTS IS THAT THEY ARE GENERALLY PREPARED WITH THE BENEFIT OF HINDSIGHT. IN ADDITION, HYPOTHETICAL TRADING DOES NOT INVOLVE FINANCIAL RISK, AND NO HYPOTHETICAL TRADING RECORD CAN COMPLETELY ACCOUNT FOR THE IMPACT OF FINANCIAL RISK OF ACTUAL TRADING. FOR EXAMPLE, THE ABILITY TO WITHSTAND LOSSES OR TO ADHERE TO A PARTICULAR TRADING PROGRAM IN SPITE OF TRADING LOSSES ARE MATERIAL POINTS WHICH CAN ALSO ADVERSELY AFFECT ACTUAL TRADING RESULTS. THERE ARE NUMEROUS OTHER FACTORS RELATED TO THE MARKETS IN GENERAL OR TO THE IMPLEMENTATION OF ANY SPECIFIC TRADING PROGRAM WHICH CANNOT BE FULLY ACCOUNTED FOR IN THE PREPARATION OF HYPOTHETICAL PERFORMANCE RESULTS AND ALL WHICH CAN ADVERSELY AFFECT TRADING RESULTS.