Sign up to save your podcasts
Or
Share FFR040 - How To Tell When A Split Test Result Is Statistically Significant
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
FFR040 - How To Tell When A Split Test Result Is Statistically Significant
- One of the most common questions I get is how do you know when to declare a winner in a split test... and also how much traffic do you need to send to it.
- One thing to keep in mind is the more traffic you can send to a split test the better... as if one page has only 12 views and the other page has 8 views you can’t call that a proper split test.
- But really the amount of hits a page has got is irrelevant, what really matters is whether the result is statistically significant or not. If it’s not then the experiment must continue until it is.
- As a general rule I won’t even look at split results unless there is a sample size of at least 100 at the very minimum. And even more so if the conversion rates are close.
- You see, just because the control or the variation has got off to flying start doesn’t mean it will continue that way. It’s quite possible that the other version will pick up steam and reserve the result.
- And terminating a split test too early because it seems obvious who has won is known as a false positive.
- So with said... the question is how can you get an accurate answer without needing to ask someone who has a PhD in Statistics.
- The answer is the VWO Statistical Significance Calculator: https://vwo.com/tools/ab-test-siginficance-calculator/
View all episodes
By Scott Dudley- One of the most common questions I get is how do you know when to declare a winner in a split test... and also how much traffic do you need to send to it.
- One thing to keep in mind is the more traffic you can send to a split test the better... as if one page has only 12 views and the other page has 8 views you can’t call that a proper split test.
- But really the amount of hits a page has got is irrelevant, what really matters is whether the result is statistically significant or not. If it’s not then the experiment must continue until it is.
- As a general rule I won’t even look at split results unless there is a sample size of at least 100 at the very minimum. And even more so if the conversion rates are close.
- You see, just because the control or the variation has got off to flying start doesn’t mean it will continue that way. It’s quite possible that the other version will pick up steam and reserve the result.
- And terminating a split test too early because it seems obvious who has won is known as a false positive.
- So with said... the question is how can you get an accurate answer without needing to ask someone who has a PhD in Statistics.
- The answer is the VWO Statistical Significance Calculator: https://vwo.com/tools/ab-test-siginficance-calculator/