Better Design Feedback

Running A/B tests

Listen Later

A/B testing is a pretty common technique in tech companies these days. That said it’s often overlooked as a form of design feedback.

A/B testing is handy. When you have enough traffic to run the test. In a reasonable amount of time. You’re testing just one single change versus another. You have the right infrastructure to run the test. And you have at least a week to run the test.

What are some common failure points for A/B testing?

  • Not having a large enough sample size. Usually you want to target at least 2000 people coming through a particular flow.
  • Not setting up a test correctly.
  • Not measuring things correctly.
  • Not letting a test run long enough. At least a week.
  • And not having a 99% statistical significance.

    • Why should I A/B test?

    The benefits of A/B testing is that you’re comparing apples-to-apples in the best possible way. You’re comparing one version to another in real time.

    A/B testing is the best. And perhaps only way to know for sure the real impact of a change that you’re going to make.

    So when should you AB test?

    1. One, when you have the traffic. If you’re a startup and you don’t have 2000 people coming through a particular flow. A/B testing might not be the best avenue for you.
    2. Two, when you have the time. Again, you probably need at least a week, and it’s helpful to run a test for a full week because you get to see weekdays and weekend.
    3. Or three, when the flow is a critical one. Onboarding flows, changes to pricing and plans, it could really be any change that has the potential to materially affect one of the metrics that you really care about.
      4. Previous
      Next – Daily journals
      ...more
      View all episodesView all episodes
      Download on the App Store
      Better Design FeedbackBy Dave Martin