Sign up to save your podcasts
Or
Share No Interference, No Ambiguity: The SUTVA Assumption ｜ EP7: Causal Inference from the Ground Up
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
No Interference, No Ambiguity: The SUTVA Assumption ｜ EP7: Causal Inference from the Ground Up
No Interference, No Ambiguity: The SUTVA Assumption
Your randomized experiment is clean. The groups are balanced and comparable. The p-value is significant. But behind the scenes, the treatment is leaking. User A shared their referral link with User B in the control group, and suddenly your "independent" comparison is contaminated.
Welcome to the most common—and most ignored—failure point in experimentation: SUTVA (The Stable Unit Treatment Value Assumption). As Fisher famously warned, consulting a statistician after a broken experiment is just asking for a "post-mortem". If SUTVA breaks, you get confident numbers that mean absolutely nothing.
In this episode, we discuss:
The Two Pillars of SUTVA: Why valid experiments require both "No Interference" (no spillovers) and "Consistency" (a well-defined treatment).
The Three Mechanisms of Interference: From direct network effects to indirect marketplace competition and systemic behavioral redirection.
The Consistency Trap: Why "the feature" can't mean different things for different users—and how to avoid "hidden versions" of your intervention.
The Experimental Fix: When to move beyond individual randomization toward Cluster Randomization, Switchbacks, or Synthetic Controls.
The Interference Diagnostic (Key Takeaways):
Contagion Risks? Randomize at the city or region level to keep social interactions within the treated unit.
Marketplace Bottlenecks? Use switchback designs to handle units competing for finite supply.
Spatial Shifts? Use buffer zones or synthetic controls to ensure you aren't just claiming credit for demand that simply moved elsewhere.
Stop running experiments that leak. Start designing for stability.
📖 Read the companion deep dive (with illustrations and takeaways): https://inferenceintel.substack.com/p/no-interference-no-ambiguity-the
About the Host
Lin Jia is a Senior Data Scientist and Craft Lead at Booking.com with over 9 years of experience. Operating at the intersection of statistical inference, causal machine learning, and GenAI evaluation, she specializes in building the frameworks that enable trustworthy, decision-ready insights under real-world constraints. A recognized expert in the field, Lin has authored research on sensitivity analysis presented at KDD 2024 and leads the development of organization-wide standards for experimentation and causal inference.
🤝 Connect with me on LinkedIn: https://www.linkedin.com/in/linjia/
🚀 Support the Craft
If you found this episode valuable, please consider:
Following the Podcast: Tap the "+" or "Follow" button on Spotify to never miss a deep dive into causal inference and GenAI.
Sharing the Episode: Know a Data Scientist or Product Leader struggling with "No Overlap"? Send this their way.
Joining the Conversation: Share your thoughts on today’s topic on LinkedIn—let’s raise the standard of the DS craft together.
View all episodes
By Lin JiaNo Interference, No Ambiguity: The SUTVA Assumption
Your randomized experiment is clean. The groups are balanced and comparable. The p-value is significant. But behind the scenes, the treatment is leaking. User A shared their referral link with User B in the control group, and suddenly your "independent" comparison is contaminated.
Welcome to the most common—and most ignored—failure point in experimentation: SUTVA (The Stable Unit Treatment Value Assumption). As Fisher famously warned, consulting a statistician after a broken experiment is just asking for a "post-mortem". If SUTVA breaks, you get confident numbers that mean absolutely nothing.
In this episode, we discuss:
The Two Pillars of SUTVA: Why valid experiments require both "No Interference" (no spillovers) and "Consistency" (a well-defined treatment).
The Three Mechanisms of Interference: From direct network effects to indirect marketplace competition and systemic behavioral redirection.
The Consistency Trap: Why "the feature" can't mean different things for different users—and how to avoid "hidden versions" of your intervention.
The Experimental Fix: When to move beyond individual randomization toward Cluster Randomization, Switchbacks, or Synthetic Controls.
The Interference Diagnostic (Key Takeaways):
Contagion Risks? Randomize at the city or region level to keep social interactions within the treated unit.
Marketplace Bottlenecks? Use switchback designs to handle units competing for finite supply.
Spatial Shifts? Use buffer zones or synthetic controls to ensure you aren't just claiming credit for demand that simply moved elsewhere.
Stop running experiments that leak. Start designing for stability.
📖 Read the companion deep dive (with illustrations and takeaways): https://inferenceintel.substack.com/p/no-interference-no-ambiguity-the
About the Host
Lin Jia is a Senior Data Scientist and Craft Lead at Booking.com with over 9 years of experience. Operating at the intersection of statistical inference, causal machine learning, and GenAI evaluation, she specializes in building the frameworks that enable trustworthy, decision-ready insights under real-world constraints. A recognized expert in the field, Lin has authored research on sensitivity analysis presented at KDD 2024 and leads the development of organization-wide standards for experimentation and causal inference.
🤝 Connect with me on LinkedIn: https://www.linkedin.com/in/linjia/
🚀 Support the Craft
If you found this episode valuable, please consider:
Following the Podcast: Tap the "+" or "Follow" button on Spotify to never miss a deep dive into causal inference and GenAI.
Sharing the Episode: Know a Data Scientist or Product Leader struggling with "No Overlap"? Send this their way.
Joining the Conversation: Share your thoughts on today’s topic on LinkedIn—let’s raise the standard of the DS craft together.