The Variance Test:
What If the Future Is a Little Worse?
A robustness test that resamples your trades but keeps only the outcomes that are worse than your backtest, so you can see whether the strategy still works with less luck.
The short answer
The Variance Test is a trading strategy robustness test created by David Bergstrom for Build Alpha. It builds a distribution from a strategy's historical trades, resamples it many times, and keeps only the simulations that perform worse than the backtest on a metric you choose, such as win rate. If a slightly worse win rate turns the strategy into a coin flip or a loser, the strategy has no margin for error and is a poor candidate for live trading.
Backtests are the best case, not the expected case
News flash: things never go as well as the backtest suggests. The strategy you are looking at was selected because its history looked good, and some of that good history was luck. Fills will be a little worse, the market will be a little different, and the win rate you see is more likely to drift down than up.
Most Monte Carlo methods reshuffle or resample your trades without bias, so they show a range of outcomes centered on the backtest. That tells you about drawdowns and streaks, but it still assumes the future looks like the past. The Variance Test asks a harder question: does this strategy still make money if it performs a little worse than it did historically?
How the Variance Test works
- Build Alpha builds a probable distribution of outcomes from the strategy's historical trades.
- It resamples that distribution, typically 1,000 times, to create hypothetical equity curves over the next N trades.
- You set how much worse each simulation must be on the metric you choose. With a 5% variation on win rate, a backtest that won 61% of the time only produces runs that win 56% of the time or less.
- Only those degraded simulations are kept and plotted, so you see the range of outcomes if the strategy loses some of its historical edge.
You control the size of the degradation, how many trades into the future to project, and how many simulations to run.
What a pass and a fail look like
- Pass: most degraded simulations still end profitable, with drawdowns you could live with. The strategy has a margin of safety.
- Fail: a small degradation pushes the simulations toward breakeven or losses. The strategy only works if the future is as kind as the past.
- Width matters: a wide fan of outcomes means results depend heavily on luck and trade order, even before degradation.
Look for strategies that hold up with slightly less optimistic metrics. If a 5% worse win rate turns your strategy into a 50-50 bet, that is not how you want to risk capital.
When the Variance Test matters most
The test is most informative for strategies that depend on a high win rate, such as many mean-reversion and short-term strategies, where a few points of win rate separate a good strategy from a bad one. It tells you less about trend-following strategies, which win rarely by design and make their money from a few large winners. For those, lean on the Noise Test, Vs Random and the Monte Carlo permutation test.
Variance Test vs other Monte Carlo tests
| Test | What it changes | What it tells you |
|---|---|---|
| Monte Carlo reshuffle | Reorders the historical trades. | How bad drawdowns and losing streaks could have been with the same trades. |
| Monte Carlo resample | Samples trades with replacement. | A range of outcomes centered on the backtest. |
| Variance Test | Resamples trades but keeps only outcomes worse than the backtest on a chosen metric. | Whether the strategy survives if the future is a little worse than the past. |
| Noise Test | Alters the price data itself and re-trades the strategy. | Whether the strategy was fitted to the noise in one price history. |
See the robustness testing guide for all 16+ tests and the order to run them in.
Where the Variance Test comes from
I created the Variance Test for Build Alpha’s first release in 2016 and walked through it on the Better System Trader podcast in March 2017. It sits alongside the Noise Test, which I also created for Build Alpha, as one of a family of robustness tests that ask how a strategy holds up when conditions are less favorable than its history. I have been writing about robustness testing for trading strategies for about ten years.
Variance Test: common questions
What is a variance test for trading strategies?
The Variance Test is a robustness test that resamples a strategy's historical trades and keeps only the simulations that perform worse than the backtest on a chosen metric, such as win rate. It shows whether the strategy still makes money if the future is somewhat worse than the past.
Who created the Variance Test?
David Bergstrom, founder of Build Alpha, created the Variance Test for the Build Alpha strategy development platform, where it shipped with the initial launch in 2016. He also created the Noise Test.
How is the Variance Test different from a Monte Carlo simulation?
Standard Monte Carlo simulations reshuffle or resample trades without bias, so outcomes center on the backtest. The Variance Test deliberately keeps only outcomes worse than the backtest, which shows how much margin for error the strategy has.
How much should I degrade the win rate?
A few percentage points is a reasonable starting point. In the example above, a 5% worse win rate took a strategy from 61% to 56% or lower and turned it into a coin flip. A robust strategy should stay profitable under a modest degradation.
Is the Variance Test useful for trend-following strategies?
Less so when degrading win rate, because trend followers win rarely by design and earn their returns from a few large winners. The Noise Test, Vs Random and the Monte Carlo permutation test are more informative for trend-following strategies.
Find out how much margin your strategy has
Run the Variance Test, the Noise Test and 14 other robustness tests on every strategy in Build Alpha.