Lore

incrementality testing

Incrementality Testing: How to Actually Measure Marketing Effectiveness (2026)

In-platform attribution only shows who converted after seeing an ad, not who converted because of it, so it over-credits some channels and under-credits others. Recast argues that geographic incrementality experiments, built on synthetic controls, are the practical way for most brands to measure what their marketing actually caused.

Recast · 2026-05-12 · English

Key ideas

  1. The video opens with eBay turning off its brand search ads. They looked wildly profitable by conventional measurement, but traffic and sales stayed flat and the change saved millions per year. The video uses this to ask whether the same test on your best channel would show real performance or nothing.

  2. Platform dashboards routinely over-claim. The example is Meta, Google and TikTok reporting $700,000 combined against $450,000 of actual business revenue. The video says this is not a bug but how in-platform attribution works, because platforms look backward, count any prior ad exposure as a win, and are incentivized to claim as many conversions as possible.

  3. Even multi-touch attribution shares the blind spot, since it tracks who saw what and not what caused the sale. A hypothetical $10,000 Meta retargeting spend reporting $25,000 in sales looks like $5,000 profit at 60% margin. If half those buyers would have bought anyway, it is a $2,500 loss.

  4. The video says the problem worsens as a brand scales. When a brand is small, attribution and incrementality are roughly the same thing. Word of mouth, press and multiple channels later make dashboards misleading, so money flows to channels that look efficient and away from ones that drive growth without getting credit.

  5. Incrementality asks a different question: would this person have converted anyway without seeing the ad? Answering it runs into the counterfactual problem, because you cannot observe both the world where the ad ran and the world where it did not.

  6. The proposed solution is borrowed from FDA drug trials: a test group that sees ads and a control group that does not, same conditions and same period. The difference in outcomes is the incremental lift, a measure of causation rather than correlation.

  7. The video ranks experiment designs by trade-off. Platform-run randomized tests are rigorous but let the platform grade its own homework. Go-dark tests are simple but confounded by time. Geographic experiments sit in the middle, work across most channels, need no platform cooperation and can be run in-house.

  8. Results must be read through the confidence interval, not the point estimate. The video says to plan around the conservative end of the range, and to be cautious if the bottom of the range is negative or break-even.

  9. Start with the channel at the intersection of high spend and high uncertainty, such as retargeting or branded search. The video stresses that tests can also show upper-funnel channels (connected TV, YouTube, podcasts, out-of-home) are under-credited, so the goal is causal truth, which sometimes means spending more.

  10. Deterministic (in-platform) attribution — The ad platform looks backward from a conversion and counts it as a win if the person saw or clicked one of its ads beforehand. Apply: Treat platform-reported revenue as a count of conversions after exposure rather than caused by it, and do not sum platform totals expecting them to match actual revenue.

  11. Multi-touch attribution — A more sophisticated attribution model that spreads credit across multiple ads instead of giving it all to the last click, but still tracks who saw what rather than what caused the sale. Apply: Use it only as a descriptive credit-splitting view, and verify channel value with experiments because it shares the same blind spot.

  12. Test vs. control incrementality experiment (FDA analogy) — A controlled experiment in which a test group sees ads and a control group does not, with the difference in outcomes being the incremental lift. Apply: Create a group that gets the ads and a comparable group that does not, run both under the same conditions and time period, and measure the outcome gap.

  13. Platform conversion lift tests — Built-in randomized control trials (Meta's conversion lift, with similar Google tools) where the platform randomly holds back ads from a subset of users and measures the difference. Apply: Use them when you want randomized rigor, while accounting for the catch that the platform is grading its own homework.

  14. Go-dark test — Turning a channel completely off and observing what happens through a simple before-and-after comparison. Apply: Use it for a quick read on a channel such as brand search, as eBay did, but be aware that seasonality, promotions and other variables can contaminate a comparison across time periods.

  15. Geographic experimentation — Randomizing by geography: increase spend in some regions, hold others constant as controls, and compare outcomes, with the control regions serving as the counterfactual. Apply: Raise spend in selected regions while keeping other regions unchanged, with no other changes in the test markets, and compare results across most channels without needing platform cooperation.

  16. Matched market testing — The traditional approach of finding two similar cities or regions (similar population, income, baseline sales) and using one as test and one as control. Apply: Increase spend in one market and hold the matched one steady, recognizing that clean matches are hard to find because of different competitors, seasonality and culture.

  17. Synthetic controls — An algorithm that uses historical data to build a weighted blend of control regions closely matching the test regions' pre-experiment behavior, serving as the counterfactual. Apply: Let a tool blend multiple control cities (like Houston, Charleston and Savannah for a synthetic New Orleans) instead of hunting for one perfect match.

  18. GeoLift test workflow (as demonstrated with Recast's GeoLift) — A five-step process: gather data, design the experiment with simulations, run it, upload post-test data, and analyze actual versus synthetic-control revenue. Apply: Upload at least 90 days (ideally a year) of daily revenue by geography, specify the spend increase, metric, expected incremental ROI, duration and budget, and pick a top-ranked configuration. Run it (e.g. +$15,000 in Austin, Boston and Denver for 4 weeks) and then analyze the gap between the counterfactual and actual lines.

  19. Confidence-interval reading rule — A rule for interpreting results by the range of likely true lift rather than the point estimate alone. Apply: Check whether the bottom of the interval still shows positive ROI, treat negative or break-even lower bounds with caution, and plan around the conservative end.

  20. High-spend, high-uncertainty first-test selection — A heuristic for choosing which channel to test first: the one that looks great in attribution dashboards but whose incrementality is doubtful. Apply: Start with candidates like retargeting or branded search, and also test upper-funnel channels such as connected TV, YouTube, podcasts or out-of-home that may be under-credited.

Insights

Attribution and incrementality converge at small scale and diverge as the brand grows, so a dashboard that was trustworthy early on can silently become misleading.

A channel that is good at finding people already deep in the purchase funnel, like retargeting, is exactly the situation where attribution over-credits.

Upper-funnel channels influence purchases days or weeks later through a different channel. Last-click style credit therefore goes to the channel that gets the clicks, and the upper-funnel channel is under-credited.

The design stage of a geo test tells you up front what the test can and cannot detect, for example an 80% chance of detecting an effect if true incremental ROI is at least 3x. If the true effect is smaller, the result may be invisible or the uncertainty too wide to be useful.

The tighter the pre-experiment fit between the synthetic control and the test markets, the more confidence the results deserve.

The ROI input to test design must be incremental ROI, not platform-reported ROAS.

A clean test requires holding everything else constant in the test markets, with no new promotions or other spend changes during the run.

The data requirement is modest: daily revenue by geography (state or DMA) for at least 90 days, ideally a year, often available from Shopify or Google Analytics 4.

«converted after seeing an ad is not the same thing as converted because of an ad.»

— 01:37

«You're trusting the platform to grade its own homework.»

— 05:48

«Always plan around the conservative end of the range.»

— 11:52

«Incrementality testing isn't just about identifying waste, it's about getting closer to causal truth.»

— 13:13

«Attribution tells you who converted after seeing an ad. Incrementality tells you who converted because of the ad.»

— 13:23

Reception

Viewers were broadly positive: they praised the clear, simple explanation of attribution versus incrementality, and one commenter pushed back mildly on how much it matters for small businesses, while several comments were off-topic spam-like account-help requests.

A clear, example-driven primer that frames the attribution-versus-incrementality distinction well and gives a concrete geo-experiment workflow. It is also a product-led pitch for Recast's GeoLift.

14:15

↳ Recast · YouTube

Watch original