Marketing mix modelling for CRM teams

The finance team commissions a marketing mix model. Eight months of work later, the results land: paid search is doing well, the brand campaign is doing better than anyone expected, and CRM has a coefficient close to zero. A meeting is scheduled to discuss the CRM budget.

The CRM lead in that meeting is usually right on the substance and completely unprepared to argue it. The programme genuinely does drive revenue, the model genuinely does not see it, and explaining why requires a conversation about collinearity that nobody in the room wants to have on a Tuesday afternoon.

We have sat on the CRM side of that meeting several times. The way to win it is to be involved eight months earlier, when the variables are being specified, rather than eight months later when the coefficients are being defended. What follows is what a CRM team needs to know to do that.

What the model measures, and why CRM fits it badly

A marketing mix model is a regression on aggregated time series. Weekly or daily sales on one side, weekly or daily marketing activity on the other, plus controls for price, seasonality, distribution, competitor activity and whatever else moves the business. It estimates how much of the variation in sales each input explains.

That design has real strengths. It needs no user-level tracking, it covers offline and brand activity, and it can be pointed at the whole business rather than the trackable part of it. As privacy constraints have tightened, this is why MMM came back into fashion after a decade of user-level attribution models being the default.

The weaknesses fall disproportionately on CRM. Three of them, in order of how much damage they do.

CRM volume tracks the business rather than driving it. You send more when there is more to say and more people to say it to, and both of those move with the trading calendar. The model sees CRM activity rising and falling with sales, which is exactly what a variable with no causal effect would also look like.

CRM effects are fast and small per message. Media effects are modelled with adstock and diminishing returns over weeks. A triggered email's effect is mostly spent within days, at a magnitude that is hard to separate from noise in a weekly aggregate.

And much of what CRM does is retention, which shows up as sales that did not fall. A model fitted to explain variation struggles with a variable whose job is to reduce variation.

The collinearity problem, stated plainly

This is the part worth being able to explain in one minute, because it is the crux of the meeting.

When two inputs move together, a regression cannot reliably tell them apart. It will allocate the shared effect between them in a way that depends on small details of the data and the specification. The split can change substantially if you add a week of data or change a control.

CRM send volume is usually correlated with the promotional calendar, with paid media bursts, and with the season. It is one of the most collinear variables in the whole model. So its estimated coefficient is unstable almost by construction, and a small or negative value is weak evidence rather than a finding.

The right response is not to claim the model is wrong. It is to ask two questions of whoever built it. What is the uncertainty interval on the CRM coefficient, and how much does it move across reasonable alternative specifications.

If the interval is wide and the estimate swings across specifications, the honest conclusion is that the model cannot resolve CRM at this level of detail. That is a very different statement from "CRM does not work", and getting it stated in those words is most of the job.

Give the model variables it can actually use

Much of the problem is fixable at specification time, and the CRM team is the only group that can fix it. Bring these to the modelling team before the data is pulled.

• Split the programme into behaviourally distinct groups. Broadcast promotional sends, triggered lifecycle messages and operational messages behave nothing alike. Modelled as one line called "CRM emails", they average into meaninglessness.
• Supply reach, not sends. Messages delivered, or better, delivered to people who have engaged recently. Send counts inflated by an unengaged file add variation that carries no effect.
• Separate promotional depth from message volume. If your sends carry discounts, the model will otherwise credit the discount to CRM or the CRM to the discount. Supply discount depth as its own variable.
• Flag the calendar explicitly. Peak weeks, sale periods and any week with an unusual send pattern. Controls that absorb the seasonality stop it being attributed to whatever moved with it.
• Provide base size as a control. Active contactable file size changes slowly and matters enormously. Without it, growth in the base is invisible and its effect lands on send volume.

Each of these reduces the collinearity a little, and together they can move the CRM estimate from uninterpretable to merely uncertain.

One thing not to bring: a request that the model be adjusted until CRM looks better. That request is remembered, and it discredits the next legitimate methodological point you make.

Use holdouts to calibrate the model, not to dispute it

The strongest position a CRM team can hold in a modelling conversation is to arrive with experimental results already in hand.

Geographic or audience holdouts give a direct read on incrementality, independent of any regression. A holdout group excluded from a programme for a defined period, compared against a matched group that received it, measures the thing the model is trying to estimate.

Modern practice is to use those results as inputs rather than as rebuttals. Experiment results can inform the model directly, tightening the range the coefficient is allowed to occupy, so the regression is anchored by measured effects instead of guessing between collinear variables. Ask whether the model is being calibrated this way, and offer the experiments if it is not.

For that to be possible, the experiments have to exist before the model is built. That means running a standing holdout at the programme level, not just per campaign, and keeping it for long enough to read a retention effect rather than a promotional one. A quarter is a reasonable minimum; a full year is better for anything seasonal.

Treat this as infrastructure rather than as a project. The holdout costs a small amount of revenue every week and it is the only asset that settles these arguments quickly.

Agree what the model is allowed to settle

The failure mode we see most is not a bad model. It is a good model used to answer questions it was never designed for.

An MMM works at the level of channel budgets over months. It is reasonable to ask it whether the CRM programme as a whole is contributing, and roughly at what scale. It is not reasonable to ask it whether a particular journey should have four messages or five, or whether Tuesday beats Thursday. Those are questions for controlled tests within the programme, of the kind we described when covering send-time optimisation, where the unit of analysis matches the unit of decision.

Write the division down before the results arrive. Which decisions the model informs, which decisions experiments inform, and what happens when the two disagree. Our preference is explicit: where a well-run experiment and a model conflict on the same question, the experiment wins, and the disagreement is recorded as a prompt to examine the specification.

Recording it is the point. This is exactly what a decision log is for, and the entry should capture what was decided, on what evidence, and what would change the answer. Eight months later, when the next model run produces a different number, that entry is the difference between a calm review and the same meeting again.

Before the next run, ask the modelling team for the current CRM specification and the uncertainty intervals, and ask whether any experiment results are being used to calibrate it. Those two questions will tell you within a week how much weight the output deserves.