Lore
Incrementality experiment design trade-offs
Three designs, ranked by trade-off:
- Platform conversion lift tests (such as Meta's conversion lift, with similar Google tools): built-in randomized control trials where the platform randomly holds back ads from a subset of users. They are rigorous, but "you're trusting the platform to grade its own homework."
- Go-dark test: turn a channel completely off and compare before and after, as eBay did with brand search. It is simple and quick, but seasonality, promotions and other variables across time periods can contaminate it.
- Geographic experiments: sit in the middle. They work across most channels, need no platform cooperation and can be run in-house. They are the video's recommended default, built on synthetic controls.
All are forms of Test vs. control incrementality experiment.