When to run MMM vs a geo holdout test

Marketing mix modeling and geo holdout tests are the two most credible tools available for measuring media effectiveness without relying on platform attribution. They are often discussed as if they are interchangeable. They are not.

Each method has a different design, a different set of trade-offs, and a different set of questions it is built to answer. Understanding which to use, and when to use both, is one of the more practically important decisions in a measurement program.

What each method actually does

Marketing mix modeling uses your historical data to build a statistical model of what drives your sales. It looks backward across a long data window, typically one to three years, and uses regression techniques to separate the contribution of each media channel from baseline factors like seasonality, distribution, and promotion. The output is a portfolio-level view of efficiency: which channels have the best ROI, where your budget is over or under-invested, and how to reallocate spend to maximise return.

A geo holdout test is a controlled experiment. You select two groups of comparable geographic markets, run media in one group and go dark in the other, then measure the difference in sales. Because you control the exposure, the causal inference is clean. The output is a direct measure of incrementality: does this channel actually cause more sales, and by how much?

Where each method is stronger

MMM is better suited to portfolio-level questions. If you want to understand how your entire media mix is performing: which channels to grow, which to cut, how much of your revenue is media-driven versus baseline. MMM gives you that full picture in a single model run. It is also the only practical option for channels that cannot be measured any other way, like television, radio, and out-of-home, where there is no pixel and no platform data to test against.

Geo holdout tests are better suited to channel-level causal questions. If you want to know with high confidence whether a specific channel (say Meta or CTV) is genuinely driving incremental sales rather than claiming credit for sales that would have happened anyway, a controlled experiment gives you the cleanest answer. The trade-off is that you can only test one or two channels at a time, and you need to accept reduced reach in your holdout markets during the test period.

When to use which

Situation	Better method
Planning annual budget across 5+ channels	MMM
Proving Meta is driving real incremental sales	Geo holdout
Measuring TV, radio, or OOH contribution	MMM
Testing a new channel before scaling budget	Geo holdout
Understanding why revenue changed quarter over quarter	MMM
CFO challenge: "prove paid social is working"	Geo holdout
Optimising budget without disrupting campaigns	MMM
Validating a surprising MMM finding	Both

The case for running both

The most rigorous measurement programs use MMM and geo holdout tests together. MMM gives you the portfolio view and the strategic allocation layer. Geo holdout tests give you causal validation of the channels that matter most or that are most questioned internally.

A common workflow: run MMM to identify your highest-spend, lowest-ROI channel. Then design a geo holdout test specifically for that channel to either confirm the finding or disprove it before making a major budget cut. If the geo holdout confirms weak incrementality, you have two independent methods pointing in the same direction, which is a much stronger case than either method alone.

This is particularly useful when an MMM finding is counterintuitive. If MMM says your biggest channel is underperforming, the natural reaction is skepticism. A geo holdout test that reaches the same conclusion removes the doubt.

The practical constraints

Geo holdout tests require geographic variation in your media buying, which not every brand has. If you are running purely national campaigns with no market-level flexibility, holdout testing is operationally difficult without a significant change to how your media is bought.

MMM requires a long enough data history and enough variation in your spend patterns to produce reliable estimates. If you have been running flat always-on budgets with little change over time, the model has limited contrast to learn from and confidence intervals will be wide.

Neither method works well in isolation indefinitely. MMM estimates drift as market conditions change. The model needs to be rerun regularly as new data accumulates. Geo holdout tests produce point-in-time estimates that may not hold as creative, targeting, and competitive conditions evolve.

The bottom line

If you are trying to decide where to allocate a media budget across multiple channels, start with MMM. If you are trying to prove or disprove the incrementality of a specific channel, run a geo holdout test. If you want the most defensible measurement program available, run both and let each method inform the other.