Better System Trader

051: Strategy evaluation techniques, flaws and solutions with Dave Walton


Listen Later

Today we’re covering a topic which can really be a concern for traders of all levels, from beginner to pro, and that is the topic of strategy evaluation.

  • Have you ever found that real-life performance does not match expected results?
  • Or perhaps you have a strategy that is stuck in a drawdown and wondering if it’s actually broken?

I’m sure we’ve all heard of data mining bias, over-optimization and curve fitting and the impacts this can have on our trading accounts.

We may be even using techniques such as Out Of Sample testing, Walk Forward Analysis, Monte Carlo analysis and a number of other measures to identify or reduce the impact of these issues, but do these approaches actually work? Are there limitations or dangers with these techniques? Are there better ways?

In this episode we talk to someone who evaluates trading systems for a living, plus his research into system evaluation techniques has won awards. The guest is Dave Walton.

Dave was the winner of the Wagner award in 2014 for a paper titled ‘Know your system – turning data mining bias to benefit through System Parameter Permutation’.

In our chat today we talk about the technique in his paper and how it can be applied to trading strategy evaluation. We also discuss some of the assumptions and limitations of the approach, and he shares with us some valuable insights he’s made since publishing the paper which have resulted in an updated approach he now considers a better alternative, so make sure you listen out for that.

Topics discussed
  • How the typical approaches to system development can introduce datamining bias without you knowing
  • The types of systems that can increase the chance of data mining bias and what to look for
  • How the method of splitting your out of sample data could be causing you to throw away good strategies
  • Out of sample, walk forward analysis and Monte Carlo - do they actually reduce data mining bias?
  • The problems with using Monte Carlo analysis to assess strategy performance and why it doesn’t protect from overfitting
  • System Parameter Permutation - how to use it, why use the median, parameter range selection and new insights since the SPP paper was published
  • How System Parameter Randomization solves some of the issues of System Parameter Permutation
  • Stochastic modelling and how it can be used to determine if a rule is adding value to your strategy

 

Disclaimer: Trading in the financial markets involves a substantial risk of loss and is not suitable for everyone. All content produced by Better System Trader is for informational or educational purposes only and does not constitute trading or investment advice. Past performance is not necessarily indicative of future results.

...more
View all episodesView all episodes
Download on the App Store

Better System TraderBy Andrew Swanscott chats with professional traders Larry Williams, Ernest Cha

  • 4.8
  • 4.8
  • 4.8
  • 4.8
  • 4.8

4.8

260 ratings


More shows like Better System Trader

View all
Michael Covel's Trend Following by Michael Covel

Michael Covel's Trend Following

670 Listeners

Freakonomics Radio by Freakonomics Radio + Stitcher

Freakonomics Radio

32,291 Listeners

Masters in Business by Bloomberg

Masters in Business

2,174 Listeners

Talking Trading - Expert trading tactics so you can excel in the markets. by Created by best-selling authors and traders - Louise Bedford and Chris Tate.

Talking Trading - Expert trading tactics so you can excel in the markets.

18 Listeners

Top Traders Unplugged by Niels Kaastrup-Larsen

Top Traders Unplugged

582 Listeners

We Study Billionaires - The Investor’s Podcast Network by The Investor's Podcast Network

We Study Billionaires - The Investor’s Podcast Network

3,369 Listeners

Chat With Traders by Ian Cox and Tessa Dao

Chat With Traders

1,987 Listeners

Desire To Trade Podcast | Forex Trading & Interviews with Highly Successful Traders by Etienne Crete

Desire To Trade Podcast | Forex Trading & Interviews with Highly Successful Traders

105 Listeners

Invest Like the Best with Patrick O'Shaughnessy by Colossus | Investing & Business Podcasts

Invest Like the Best with Patrick O'Shaughnessy

2,317 Listeners

The Jordan B. Peterson Podcast by Dr. Jordan B. Peterson

The Jordan B. Peterson Podcast

34,127 Listeners

ChooseFI by ChooseFI

ChooseFI

5,088 Listeners

Flirting with Models by Corey Hoffstein

Flirting with Models

225 Listeners

Risk of Ruin by Half Kelly Media

Risk of Ruin

231 Listeners

The Algorithmic Advantage by The Algorithmic Advantage

The Algorithmic Advantage

11 Listeners

Words of Rizdom by Riz Iqbal

Words of Rizdom

16 Listeners