Free Friday 8 – Randomized Monte Carlo
Randomized Monte Carlo Strategy Analysis
In this Free Friday, I want to introduce a new feature to Build Alpha: The Randomized Monte Carlo Test. The purpose of this test is to give us another robustness test or tool to identify potentially unreliable trading strategies or lying backtests.
As system traders, all we can do is improve the probabilities that the strategies we take live will succeed. Identifying lying backtests or misleading results can both save the trader money and headaches.
The strategy below shorts the EURUSD futures contract and has shown nice results on both in-sample and out-of-sample testing from 2002 to 2017 with a fairly large sample size of over 600 trades. Should we greenlight this strategy and trade it live?
All performance metrics and even the simple reshuffle monte carlo method (pictured below) seems to give favorable insights to trading this strategy live. This strategy shows almost all Monte Carlo paths are profitable after 150 trades. Should we go live?
Let’s introduce one more test, the Randomized Monte Carlo test. This test can help us determine if this strategy’s backtest is lying to us and can potentially harm our account.
What is the Randomized Monte Carlo Test?
The Randomized Monte Carlo test re-trades the entry signal while modifying each trade’s exit at random (and within reason). The Randomized Monte Carlo test will not introduce new exit logic or remove any exit logic. That is, if the original strategy does not use a stop-loss then the Randomized Monte Carlo test will not introduce one. This is vital as altering the strategy’s logic can alter the underlying strategy to the point where the results do not tell us anything informative about the original strategy.
This particular strategy uses three unique exit strategies:
- Maximum 3-day hold || First profitable close || Stop at rolling 5-day high
In the Randomized Monte Carlo test, trade one will enter as expected but we may exit with a two-day max hold, first profitable close, or stop at the rolling four-day high.
The second trade will enter as expected but we may exit with a three-day max hold, second profitable close, or a stop at the rolling five day high.
Trade three will enter as expected but we may exit with a three-day max hold, first profitable close, or stop at the rolling seven day high.
This process will repeat for ALL the trades randomizing the exit each time. The first three hypothetical trades of the first run, that are stated above, are summarized below for simplicity and comparison’s sake. Again, this is random so you can imagine the variations possible.
- Maximum 2-day hold || First profitable close || Stop at rolling 5-day high
- Maximum 3-day hold || Second profitable close || Stop at rolling 5-day high
- Maximum 3-day hold || First profitable close || Stop at rolling 7-day high
We then repeat this test and re-trade the strategy 1,000 or more times to make sure we have a reliable sample and remove any lucky “randomizations”. The results of this test on this Free Friday strategy are below. Yikes.
What does the Randomized Monte Carlo Test tell us?
In this particular example, this test and new feature in Build Alpha was able to identify a lying backtest and save us from implementing a false positive or “too good to be true” trading strategy.
The test is intended to validate we have not overfit our exit strategies/signals. If the Randomized Monte Carlo results are all positive, then it can tell us that our strategy’s entry has enough edge worth harvesting and could maintain profitability even with random exits. However, seeing struggling results like the graph above indicate a lying backtest and potentially overfit exits. This is just another tool in our trading toolbox to help prevent or reduce the risk of curve fitting.
For instance, strategies showing nice equity curves but poor edge ratios are always suspicious. The Randomized Monte Carlo test is a good candidate here and can identify overfit exits that are squeezing out profits when the edge ratio indicates there should not be much! Anyways, more to come in Build Alpha on strategy robustness and reliability.
8 thoughts on “Free Friday 8 – Randomized Monte Carlo”
What exactly does radomized exits mean? Exit after x days?
What exactly does radomized exits mean?
Exit after x days? Where x is a random number of days for each trade?
Yeah, exactly. It is a simple test to check if we have discovered curve fit exits or an actual edge in regards to our entry signal(s).
The length of the position hold may have a material effect on the trading system though, don’t you think? As an extreme example, someone attempting a momentum trade based on fundamental changes to the stock, but then exiting the same day, will show poor returns. This sort of trade is dependent on a long term hold. Likewise, someone developing a 24 hour trade would experience poor returns if randomizing the length of position hold for too long.
I think the key is that returns should show a smooth variation around the optimal exit delay, rather than a spike. If five days is optimal, four and six days ought be reasonable as well. If a four and six day hold kill the system, then a five day hold is likely curve-fit. the randomization of exits should be within very strict ranges that are applicable to the trade type.
Great blog btw. I’ve added it to my reading list.
You’re absolutely right in that certain tests lend themselves better to certain systems; this is certainly where the human is still important in the automated trading process.
Also, this is just one of the tests available at our disposal. The test you’re describing is more similar to Build Alpha’s “Delayed Entry and Exits” test. This test shows a variety of results if you were to delay either or both the entry or exit by -3 to +3 bars. This would cover/accomplish the 4-6 period test with original exit at 5 as you described, for example (and more). I really appreciate you reading and the words of praise. Thanks for taking the time!
I think there is an error here:
The first part of your entry Open(0) does not match up with the MC Analysis which say Open (3)?
Could you double check the formula.
Lol obviously we all make type-os!
The formula in the excel table should read the same as the Build Alpha charts (which are correct)
Close>Low and High>Close and Open<=Open and Open<=High Thanks, David
That should be Close(0) and Close(3)