Yue Meng is the Senior Data Scientist at Delivery Hero, an on-demand delivery platform. Previously she was a marketing consultant for Ninah Consulting. She has a master’s degree in statistics. She is based in Berlin, Germany. 

Questions Yue Answered in this Episode:

• What methodologies do you use to evaluate your various campaigns? 
• What are your plans for A/B testing once device IDs are no longer available with the data privacy changes to iOS14? 
• Can you give us an example of how you would analyze an offline campaign?
• How do you know if your baseline model is accurate enough?
• How would you improve your model if it wasn’t accurate enough?
• Can you use the control group only to focus testing? 

Timestamp:

• 1:02 Yue’s background
• 4:45 Why Delivery Hero trusts A/B tests
• 7:54 Plans for marketing campaign testing without device IDs
• 11:29 KPIs for evaluating offline campaigns
• 15:10 Measuring uplift from offline campaigns with time series
• 21:39 Calculating accuracy with baseline models
• 24:30 Using geo as a proxy for user-level data
• 31:34 Looking ahead at testing solutions

Quotes: 

(9:21-9:43) “This offline scenario is very similar to an online scenario without the device ID, I would say. So probably some models that we’re using for the offline campaigns estimation can be used on the online campaigns also. That’s what we are currently trying to do.”

(17:07-17:20) “By using the time series analysis, the simplest model you can use, you’re not adding any other factors or any other features. You only focus on the time series by itself.”

(24:30-24:41) “Now the control group on a user level is impossible. What we can try to do as a proxy is to use instead of user-level to use geo-level, for example, the city level.” 

Mentioned in this Episode:

• Yue Meng’s LinkedIn
• Delivery Hero