Marketing Mix Modeling You Can Check: Adstock, Saturation and ROAS Against a Known Truth

The problem with grading a marketing mix model

A marketing mix model reads a few years of weekly spend and sales and tells you what each channel returned. The trouble is that real data never comes with an answer key. If the model says paid social returns $0.97 per dollar, there is no ledger to check it against, so a confident wrong answer looks exactly like a confident right one.

The way around that is the same one engineers use for any estimator: run it where the truth is known. I did that with two open-source libraries I maintain. Papilon is a Bayesian marketing mix model. PyCausalSim is a causal simulator, and its new marketing module generates whole populations of brands from a declared causal graph, each with its own answer key. Papilon does the estimating; PyCausalSim builds the worlds it is graded in and plays its budget plans forward week by week.

The short version: the model's channel estimates were honest, with 90% ranges that held the true return 94% of the time. The budget plans built on those estimates were not. The standard approach, moving money to maximize expected sales, captured 32% of the gain that was available and lost money for a third of the brands. On brands with product launches the model wasn't told about, it lost money 83% of the time. Three changes fixed most of it, and they are now the defaults in Papilon 0.4.

You get the model in two formulas, runnable Python for one brand and for the benchmark, what went wrong, the three fixes with their numbers, and where it still breaks. Everything on the single brand can be explored in the live Papilon demo.

The model in two formulas

Weekly sales are a baseline plus a contribution from each channel, plus price, promotions and noise:

The fraction is the saturation curve, the Hill form with shape 1: zero at zero spend, half of the channel's maximum weekly effect at spend K, and flattening after that. β is the most a channel could add in a week, and K, the half-saturation point, says how quickly it gets there. This is where diminishing returns live.

The term a is adstock, the spend that is still working. Advertising doesn't stop at the end of the week it runs, so a is a weighted sum of recent spend:

Geometric carryover peaks the week money is spent and decays by α each week after. Delayed carryover peaks θ weeks later, which fits video and other brand media that people see, remember and act on later. Both forms come from Jin and colleagues at Google, whose 2017 paper set the template most open-source models still follow.

One brand: the estimates pass, the plan doesn't

The first test is a single simulated brand from Papilon's make_synthetic_mmm(): 156 weeks, about $868,000 in weekly sales and $149,000 in weekly media across paid search ($30,300), paid social ($49,200), online video ($47,200) and display ($22,400). Paid search spend follows seasonal demand, the way it does for any brand bidding on searches people make anyway, and online video's effect peaks two weeks after it airs.

Papilon compared carryover shapes before estimating anything. The data favored delayed video by a log Bayes factor of 18.8, a posterior probability above 99.9%, and put the peak at 2.0 weeks. The truth is 2. That check matters: forced to assume every channel peaks the week it runs, the same model credits video with a return of 3.38 against a true 1.57, more than double, with a range that excludes the truth.

With the right shape, the estimates against the answer key:

Every true value is inside its range. Paid search has the widest range because its spend moves with the season and the data can't fully separate the two. That is the honest answer.

Then the plan. Asked to move the same $149,200 a week to maximize expected sales, with each channel free to go anywhere from zero to three times its history, the model cut paid search to $15,300, paid social to $38,000 and display to $11,700, and moved the money into video, up to $84,200. It expected the plan to add $12,500 a week. Against the true curves it added $1,100. The best possible reallocation of that budget was worth $5,600.

It gets worse in the first quarter. PyCausalSim can take Papilon's fitted curves and play a plan forward week by week, with carryover and the ramp from the old mix to the new one. Under the true curves the plan loses $28,500 of sales over its first 13 weeks: the cuts to search and social cost money immediately, while video's extra effect arrives weeks later. The model expected a gain of $129,700.

"""mmm_check.py: fit, plan two ways, and play both plans forward a quarter (Brian Curry | Vector 1 Research)."""
import papilon as pp                                   # 0.4
from pycausalsim.futures import compare_plans, media_plan_scm

MEDIA = ["paid_search", "paid_social", "online_video", "display"]
df, truth = pp.make_synthetic_mmm()                    # 156 weeks, $ thousands, with the answer key

model = pp.MMM(data=df, target="sales", media=MEDIA, date_col="date",
               controls=["price_index", "promo"], yearly_seasonality=2, seed=11)
model.compare_carryover({"all_geometric": [], "video_delayed": ["online_video"]})
print(model.get_roi().round(2))

hist = df[MEDIA].mean()
old = model.optimize_budget(total=hist.sum(), objective="mean", max_change=None)  # maximize expected sales
new = model.optimize_budget(total=hist.sum())          # 0.4 default: protect the downside, +/-50% per channel
print(new[["current_spend", "optimal_spend", "outside_support"]].round(1))
print(new.attrs["summary"])                            # expected gain, 90% range, chance it beats the current mix

# Grade both plans against the answer key (steady state, $K a week)
tc = truth["channels"]
true_weekly = lambda x: sum(tc[c]["beta"] * v / (v + tc[c]["half_sat"]) for c, v in zip(MEDIA, x))
for name, p in [("old rule", old), ("0.4 default", new)]:
    print(name, "true gain %.2f" % (true_weekly(p.optimal_spend) - true_weekly(p.current_spend)))

# The first quarter, week by week, through 2,000 joint posterior draws
scm = media_plan_scm(model.posterior_curves(), hist.values, weeks=13, draws=2000, seed=1)
plans = {"current": hist.values, "old rule": old.optimal_spend.values, "0.4 default": new.optimal_spend.values}
print(compare_plans(scm, plans).round(1))

One brand is an anecdote, though. Maybe this one was unlucky.

120 brands from PyCausalSim

PyCausalSim's marketing_worlds() simulates many brands at once, one per draw of a temporal causal model. Each brand gets its own spend levels, true returns between 0.5 and 2.2, carryover and saturation, so the benchmark covers brands where search is the best channel and brands where it is the worst. Seasonal demand drives both paid search spend and sales in every brand.

Half the brands are standard. The other half are hard: two to four product launches lift search and social spend and demand at the same time, and social creative loses 20% to 40% of its effect over the three years. Launches are the confounder every real marketing mix model faces, because brands spend more when they expect to sell more.

For each brand, Papilon is fitted, plans the same weekly budget under several rules, and each plan is graded against the truth: the share of the best possible gain it captured, and whether it lost money.

from pycausalsim.futures import marketing_worlds

worlds = marketing_worlds(60, hard=True, seed=7)       # 60 brands, one per draw, built in about 0.1 seconds
df, truth = worlds.frame(0), worlds.truth(0)           # what a modeler sees, and the answer key
truth["channels"]["paid_search"]                       # true ROAS, marginal ROAS, carryover, saturation

The full benchmark, with every rule and fit, is benchmarks/recovery.py in the Papilon repository. Each fit takes a few seconds, after a 0.4 change that made fitting 11 times faster.

What the benchmark found

The estimates held up on standard brands. Across 60 of them, the 90% ranges held the true return 94% of the time, which is what a calibrated model should do.

The plans did not. Maximizing expected sales captured 32% of the available gain on average, lost money for 33% of brands, and promised about four times what it delivered: $8,400 a week expected against $2,100 realized. The rule leans hardest on whichever channel the model happens to overrate, and with four channels there is almost always one.

The hard brands broke the estimates too. With launches hidden from the model, paid search was overcredited by 3.46 in ROAS on average, and its 90% range held the truth for only 8% of brands. Plans built on that lost money for 83% of brands and destroyed more value than the best plan could have added.

Three fixes, and what each one bought

1. Protect the downside instead of maximizing the average. Papilon 0.4 plans by maximizing the 10th percentile of the gain over the current mix, so a plan has to hold up in the scenarios where the model is wrong, and it keeps each channel within 50% of its current spend. On standard brands that captured 46% of the available gain and lost money for 7% of brands, against 32% and 33% for the old rule. With launches hidden it still captured little, but it broke even on average where the old rule lost heavily. Plans also flag any channel pushed beyond the highest spend the brand has sustained, and report the chance they beat the current mix.

On the single brand above, the new default moved $22,600 instead of $36,900 and added $3,900 a week against the truth, about 70% of the best possible gain. Over its first quarter it gained $19,400 where the old rule lost $28,500. Both plans dip in the first weeks while video's extra effect builds; the new one is back to even by week 8, while the old one, recovering about $1,100 a week, would need roughly nine months.

2. Tell the model what you already know. A brand knows its own launch calendar. Entered as a control, it took paid search's range from holding the truth 8% of the time back to 95%, and the share of hard brands where the new default lost money fell from 45% to 12%.

3. Calibrate the confounded channel with an experiment. A simulated geo lift test on paid search, measured with realistic noise and entered as roi_priors={"paid_search": (measured, 0.25)}, cut search's average error from 0.87 to 0.16. With the test, the new default captured 58% of the available gain and lost money for only 2% of standard brands; with the launch calendar too, hard brands reached 50% with losses at 8%. Google's Meridian documentation recommends the same calibration with incrementality experiments.

The new default gives up a few points of gain in the easy cases to avoid the expensive ones. Uncapped plans did slightly better on average once the model had the launch calendar and a lift test; nothing did well without them.

Where it breaks

The simulated worlds are kinder than real data in three ways.

The functional form is right. The data was generated with the same saturation curve and carryover kernels the model fits. Real response curves can be S-shaped or kinked, and a model with the wrong curve can fit history well and extrapolate badly, which is exactly what a reallocation asks it to do.

The confounders are the ones I planted. Seasonality and launches are in the graph because I put them there. Real brands have competitor moves, distribution changes and sales pushes, and each one the model doesn't see gets credited to media.

Display is overrated. Across every setting, Papilon overestimated display's return by about 0.5 on average, most likely because its priors are generous for a small channel with a weak signal. That is the next thing to fix.

The single brand also shows the limit of the ranges. Both plans' first-quarter results fell below the model's 90% quarter ranges there, because the model underrated search and social and overrated video. Across the benchmark the quarter ranges held the truth about nine times in ten on standard brands, but on any one brand they can miss.

The path forward

Grade your model before you trust it. Fit your setup to simulated brands that look like your business, with known returns, and check that its ranges hold the truth about nine times in ten. PyCausalSim's marketing_worlds() builds them from a graph you declare.

Choose carryover shapes with the data. A wrong shape can double a channel's credit while making the model look more certain.

Put known events in the model. Launches, big promotions and distribution changes belong in the controls, not in the residual where media can absorb them.

Run a lift test on the channel the data can't separate from demand, and feed the result back as a prior. For most brands that is paid search.

Plan for the downside. Maximize what the plan delivers when the model is wrong, cap how far any channel moves, read the chance the plan beats the current mix, and play the plan forward a quarter to see the ramp before the budget shifts.

You can compare both rules in the live Papilon demo: it shows each channel's estimate against the truth, the carryover comparison, and a budget planner with Papilon's 0.4 plan, the older rule and the best possible plan side by side. The planner runs on 400 posterior draws in your browser, so its numbers can differ from the script's by a few hundred dollars a week.

Sources

• Jin, Wang, Sun, Chan and Koehler, Bayesian Methods for Media Mix Modeling with Carryover and Shape Effects, Google, 2017
• Chan and Perry, Challenges and Opportunities in Media Mix Modeling, Google, 2017
• Chen, Chan, Perry, Jin, Sun, Wang and Koehler, Bias Correction For Paid Search In Media Mix Modeling, Google, 2018
• Google, Meridian: Channel calibration recommendation and score
• Vehtari, Simpson, Gelman, Yao and Gabry, Pareto Smoothed Importance Sampling, Journal of Machine Learning Research, 2024
• Papilon, github.com/Bodhi8/papilon, version 0.4, and PyCausalSim, github.com/Bodhi8/pycausalsim, version 0.4
• Vector 1, Papilon demo and the benchmark code in this article