Sign up to save your podcasts
Or
Share How Long Should You Run a Split-test? | Ep. 58
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
How Long Should You Run a Split-test? | Ep. 58
here is the resource to test sample size:
https://www.abtasty.com/sample-size-calculator/
How long do you run an A/B test?
The discrepancy between the two versions is gradually eroded as time goes by – two weeks after the starting point there’s hardly any observable difference at all.
This phenomenon where the results converge is a typical situation: the modification made does not have a real impact on conversion.
Rules on how long to run a test:The statistical confidence level (95% or more)
The size of the sample
The representativeness of your sample
The test period and the device being tested
Your business circleSeasonality of your business.Your audience (email, repeat buyers, sales, loyalty members)
Pro Tip: The lower the expected improvement rate, the greater the sample size needed to be able to detect a real difference.
Pro tip: small sites under 100k visitors a month: Test bigger things, like bigger site overhauls so the conversion rate or AOV uplift is higher. That way there is a higher chance you can trust the data.
Bigger sites can get away with testing smaller things because they got more data to rely on.
Pro Tip: by default 2 weeks is going to be a good length for a test but don't be afraid to extend it.
Pro Tip: Do not pause it, and start again or change other variables. If you got a bug, start over completely.
Pro tip: exclude your own IPs and employees too, to not mess up the test data.
Finally don’t forget that A/B testing without perfect statistical significance is still much better than implementing based on subjective opinions!
View all episodes
By Daniel Chaberthere is the resource to test sample size:
https://www.abtasty.com/sample-size-calculator/
How long do you run an A/B test?
The discrepancy between the two versions is gradually eroded as time goes by – two weeks after the starting point there’s hardly any observable difference at all.
This phenomenon where the results converge is a typical situation: the modification made does not have a real impact on conversion.
Rules on how long to run a test:The statistical confidence level (95% or more)
The size of the sample
The representativeness of your sample
The test period and the device being tested
Your business circleSeasonality of your business.Your audience (email, repeat buyers, sales, loyalty members)
Pro Tip: The lower the expected improvement rate, the greater the sample size needed to be able to detect a real difference.
Pro tip: small sites under 100k visitors a month: Test bigger things, like bigger site overhauls so the conversion rate or AOV uplift is higher. That way there is a higher chance you can trust the data.
Bigger sites can get away with testing smaller things because they got more data to rely on.
Pro Tip: by default 2 weeks is going to be a good length for a test but don't be afraid to extend it.
Pro Tip: Do not pause it, and start again or change other variables. If you got a bug, start over completely.
Pro tip: exclude your own IPs and employees too, to not mess up the test data.
Finally don’t forget that A/B testing without perfect statistical significance is still much better than implementing based on subjective opinions!