u/michael-recast

We graded four big (and free) geo-testing tools against known answers across 32,000 simulated experiments. We found that there’s no clear "best one", as each one fails in its own way, and that the direction of the failure is what should help you decide which tool to use.

We did this study because if you run geo-based experiments (turning your ads on in some cities, off in others, then measuring the lift), the result you get at the end comes from a statistical tool, usually CausalPy, Meta GeoLift, Google Matched Markets, or CausalImpact. Marketing teams tend to treat that number (the output from the tool) as the truth, and normally there's no way to check it.

So we built simulated daily sales data where we already knew the answer: in each version we either planted a 7.5% incremental sales lift from the campaign or no impact at all (0% lift). We then ran all four tools on the same thousand variations, giving us an answer key that we could grade them against.

What came back was that every tool has its own "personality”, and you should pick yours accordingly.

• Meta GeoLift is what you’d call “the skeptic”. It almost never declares a fake winning campaign (3 to 5% of the time), but it labels about 9 in 10 real winning campaigns as "inconclusive." It's safe against false positive results but frustrating as it misses real positive ones.
• CausalImpact is “the optimist”. It catches most real wins, but it also declares a win on no-campaign-effect data about 30% of the time.
• CausalPy and Google Matched Markets are somewhere in the middle on both counts.

So the question isn't "which tool is best”, but which mistake costs your business more: scaling budget into a channel that does nothing, or failing to scale a channel that actually works? If you can answer that, the tool choice mostly makes itself.

There were also two practical warnings from the study. First, don't run a geo experiment off one month of historical data. With 30 days of pre-test data instead of 90, no tool gave a reliable read.

Second, don't run tests in your biggest market just because leadership wants it. A treated market 5x bigger than the controls blew up the uncertainty in the results output for all four tools.

If the results sound interesting they're on the recast site and happy to share them directly (unable to link here).

Our research team ran a simulation study and found that the four big open-source geo-experiment tools (CausalPy, Meta GeoLift, Google Matched Markets, and CausalImpact) recover almost the same point estimate on the same data, then disagree about whether that estimate is significant. Since the disagreement lives in the uncertainty (not in the point estimate) the tool you pick may determine which error you ship.

In a "live" experiment you can't grade the tool because we don't know what ground truth is. The counterfactual is unobservable  so "is this lift real?" has no answer key. That's why we had our research team generate 8,000 synthetic daily-sales panels, each with either a 7.5% multiplicative lift on the treated geo or no effect at all (0% lift). They ran all four tools on the same panels and scored every fit against the planted truth, so there were 32,000 fits in all across four scenarios.

Across the non-outlier scenarios, every tool recovered the 7.5% lift within a few percentage points, so judged on point estimates alone they look interchangeable. The split is entirely in how they handle uncertainty: coverage (how often the 95% interval actually contains the true effect) and power (how often it detects a real effect at all). On those two axes the tools fall into three camps:

• Meta GeoLift is the most cautious with coverage of 92–95% and a false positive rate of 3–5%. It failed to reject zero in 89–96% of runs where a true 7.5% lift was present.
• CausalImpact is the opposite with the most power of the four (false negative rate 34–48%), but coverage of only 70–72%, a false positive rate of 28–30%, and a consistent upward bias of +1.87 to +4.21 percentage points that shifts the whole interval high.
• CausalPy and Google Matched Markets sit between them with coverage of 76–86%, false positive rates of 14–25%, meaning they’re both under-covered and under-powered at the same time.

There are four things from the study I'd take back to a measurement program:

1. Read coverage and power together: A tool can keep its 95% coverage promise and still be useless for detection. GeoLift holds about 95% coverage in the short-history scenario while missing the real effect 95.7% of the time.
2. Pick the estimator whose error profile matches the cost asymmetry of your decision and not the one with the best-looking single metric.
3. Scarce history sharpens each tool's failure mode. Cutting the pre-period from 90 days to 30 didn't degrade the tools uniformly. The decisive ones threw more false positives (above 24%), the cautious one climbed to a 95.7% miss rate.
4. Test-market design beats estimator choice. When the treated geo was 5x the size of the median control, every tool's intervals widened 4–5x and most overestimated the lift by 2–4 percentage points. No estimator compensates for a structurally hard design.

We made everything reproducible including the data-generating process, seeds, configs, per-iteration results, and a Makefile that runs the whole pipeline. The generator is parameterized, so if you think it should be harder (idiosyncratic geo trends, heavier tails, spillovers between markets) those are exactly the runs I'd like to see.

If you’re interested in the full study + code, you can find both here:

• Code: https://github.com/getrecast/geolift-simulation-study
• Full report: https://research.getrecast.com/geolift-sim-study

edited: fixed the code link to the public repo