I've seen it happen time and time again—marketers compare performance across one partner to the next, but when looking at incremental lift, things can often look too good to be true. Upon digging in, more times than not, the control group used in the incrementality test wasn't actually comparable.
If you're running an incrementality test, choosing the right control group is critical. Pick the wrong one, and your lift results could be completely misleading.
Let's take In-N-Out as an example. It's only in four states. If In-n-Out's marketing team ran a Randomized Control Trial (RCT) using people throughout the U.S. as the control group, of course, they'd see huge lift—because someone in Wisconsin, New York, or Florida is far less likely to walk into a Southern California location. But does that actually tell you if your marketing worked? No—it's just bad methodology.
✅ The key to an accurate lift test? Your control group must be as SIMILAR as possible to your test group.
The 3 Main Types of Control Groups & Their Trade-Offs
1️⃣ Randomized Control Trials (RCTs) 🎯
People are randomly assigned to test or control.
Pros: The historical Gold standard for causal measurement, eliminates selection bias.Cons: If not carefully designed, can create control groups that aren't truly comparable (like our In-N-Out example).
2️⃣ Public Service Announcement (PSA) Holdout 🎗️
Instead of suppressing ads for the control group, they're shown a neutral PSA ad (like an anti-smoking campaign).
Pros: Ensures the control group has a similar ad experience, avoiding exposure bias.Cons: Labor-intensive and financially expensive—you're allocating precious media spend to creative that isn't even your own. Plus, it doesn't perfectly isolate the ad's impact, since both groups are still being served media.
3️⃣ Ghost Bidding 👻
Fact: 80% of all auctions are lost—Ghost Bidding leverages the users who you've targeted but never won an ad.
Pros: Keeps everything identical—same audience, same placements, same media dynamics. The new Gold standard. 🏅Cons: More complex to execute and requires demand-side platform (DSP) support & tech.
Why Similarity in Control Groups Matters🔹 If your control group isn't exposed to the same purchase opportunities, your lift results will be artificially inflated.🔹 The closer your test and control groups are in demographics, geography, and behavior, the more reliable your incrementality results will be.🔹 The best approach depends on your campaign, but bad control groups = bad insights.
If your control group isn't truly comparable to your test group, you're not measuring lift—you're measuring a flawed experiment. 🏆
#Incrementality #LiftStudies #MarketingMeasurement #AdTech #DigitalMarketing #RCT #MarketingScience #Advertising #GrowthMarketing