Through online optimization testing, we strive to discover which webpage or email message will perform best by trying each version with a random sample of target prospects.We decide during test design how much difference we must see to warrant making a change, and what level of confidence we need to consider the data "valid" using statistical methods.

But is sample size the only factor that should be considered when assessing the validity of our test results? In this clinic, we will identify the four greatest threats to test validity, and look at some case studies that reveal the dangers of relying on a "statistically valid" sample alone.