Sign up to save your podcasts
Or
Share Comparing Apples to Apples: The Exchangeability Assumption | EP5: Causal Inference from the Ground Up
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Comparing Apples to Apples: The Exchangeability Assumption | EP5: Causal Inference from the Ground Up
Comparing Apples to Apples: The Exchangeability Assumption
Your dashboard flags a troubling trend: users who contacted customer support have a 40% higher churn rate than those who didn’t. The immediate takeaway seems obvious—support is failing.
But is it? Or did those customers contact support because something had already gone wrong?
In this episode, we tackle the heart of the "Bad Comparison" problem. We dive into Exchangeability—the fundamental assumption that allows us to treat observational data as if it were a randomized experiment. If your groups aren't exchangeable, your model isn't measuring an effect; it's just measuring a pre-existing difference.
In this episode, we discuss:
- The Support Trap: Why correlation often hides the "underlying fire" and leads to backward business decisions.
- The "Swap" Test: A simple mental framework to determine if your treatment and control groups are truly comparable.
- Bias Under the Null: How a model can show a massive "effect" even when the treatment does absolutely nothing.
- Forcing Exchangeability: The role of conditioning and why choosing the right covariates is the most critical decision a Data Scientist makes.
Stop settling for bad comparisons. Start ensuring your data is "exchangeable" before you trust the result.
📖 Read the companion deep dive (with illustrations and takeaways): https://inferenceintel.substack.com/p/comparing-apples-to-apples-the-exchangeability
🚀 Support the Craft
If you found this episode valuable, please consider:
- Following the Podcast: Tap the "+" or "Follow" button on Spotify to never miss a deep dive into causal inference and GenAI.
- Sharing the Episode: Know a Data Scientist or Product Leader struggling with the "Data Validity Cliff"? Send this their way.
- Joining the Conversation: Share your thoughts on today’s topic on LinkedIn—let’s raise the standard of the DS craft together.
About the Host
Lin Jia is a Senior Data Scientist and Craft Lead at Booking.com with over 9 years of experience. Operating at the intersection of statistical inference, causal machine learning, and GenAI evaluation, she specializes in building the frameworks that enable trustworthy, decision-ready insights under real-world constraints. A recognized expert in the field, Lin has authored research on sensitivity analysis presented at KDD 2024 and leads the development of organization-wide standards for experimentation and causal inference.
🤝 Connect with me on LinkedIn: https://www.linkedin.com/in/linjia/
View all episodes
By Lin JiaComparing Apples to Apples: The Exchangeability Assumption
Your dashboard flags a troubling trend: users who contacted customer support have a 40% higher churn rate than those who didn’t. The immediate takeaway seems obvious—support is failing.
But is it? Or did those customers contact support because something had already gone wrong?
In this episode, we tackle the heart of the "Bad Comparison" problem. We dive into Exchangeability—the fundamental assumption that allows us to treat observational data as if it were a randomized experiment. If your groups aren't exchangeable, your model isn't measuring an effect; it's just measuring a pre-existing difference.
In this episode, we discuss:
- The Support Trap: Why correlation often hides the "underlying fire" and leads to backward business decisions.
- The "Swap" Test: A simple mental framework to determine if your treatment and control groups are truly comparable.
- Bias Under the Null: How a model can show a massive "effect" even when the treatment does absolutely nothing.
- Forcing Exchangeability: The role of conditioning and why choosing the right covariates is the most critical decision a Data Scientist makes.
Stop settling for bad comparisons. Start ensuring your data is "exchangeable" before you trust the result.
📖 Read the companion deep dive (with illustrations and takeaways): https://inferenceintel.substack.com/p/comparing-apples-to-apples-the-exchangeability
🚀 Support the Craft
If you found this episode valuable, please consider:
- Following the Podcast: Tap the "+" or "Follow" button on Spotify to never miss a deep dive into causal inference and GenAI.
- Sharing the Episode: Know a Data Scientist or Product Leader struggling with the "Data Validity Cliff"? Send this their way.
- Joining the Conversation: Share your thoughts on today’s topic on LinkedIn—let’s raise the standard of the DS craft together.
About the Host
Lin Jia is a Senior Data Scientist and Craft Lead at Booking.com with over 9 years of experience. Operating at the intersection of statistical inference, causal machine learning, and GenAI evaluation, she specializes in building the frameworks that enable trustworthy, decision-ready insights under real-world constraints. A recognized expert in the field, Lin has authored research on sensitivity analysis presented at KDD 2024 and leads the development of organization-wide standards for experimentation and causal inference.
🤝 Connect with me on LinkedIn: https://www.linkedin.com/in/linjia/