In this episode, we discuss Expedia’s need to evaluate many ranking ideas quickly so that A/B testing would not become a bottleneck. We explore their three-stage experimentation funnel: backtesting to remove weak candidates, interleaving to rapidly compare promising ones, and A/B testing to validate final business impact. What made this design approach effective was using the right evaluation tool at the right step instead of forcing A/B testing to do everything.
For more details, you can refer to their published tech blog, linked here for your reference: https://medium.com/expedia-group-tech/interleaving-for-accelerated-testing-75adc644027b