Holdout groups, or you cannot prove CRM works

A finance director asks a reasonable question in a quarterly review. The CRM slide shows a large share of online revenue attributed to email and push. She asks what would have happened if none of it had been sent.

The honest answer, in most programmes, is that nobody knows. Attribution reports on who bought after receiving a message. It says nothing about who bought because of it, and the difference between those two numbers is the entire value of the channel.

There is one reliable way to answer her. Keep a group of customers out, treat them identically in every other respect, and compare. A holdout group is not a sophisticated technique, and that is the point: it is the cheapest experiment available and the only one that measures what the channel adds.

Attributed revenue is not incremental revenue

CRM attribution is biased upwards by construction. The people most likely to receive and open a message are the people most engaged with the brand, and engaged customers buy more whether or not you mail them.

Last-click makes this worse, because an email click is a cheap final touch on a journey that was already going to end in a purchase. The reporting rebuild that followed the move away from Universal Analytics exposed a lot of these numbers, since a change in attribution model moved totals that were supposed to be facts.

Incrementality is the number that survives that scrutiny. It is the difference in behaviour between customers who could receive the programme and otherwise identical customers who could not.

Every other approach estimates it. A holdout measures it directly, and it does so in units the finance team already accepts.

Choose the level before you choose the size

Holdouts operate at three levels, and they answer different questions. Most programmes need all three eventually, and should start with one.

• Campaign holdout: a small random share excluded from a single send. Answers whether that campaign earned its place. Cheap, fast, noisy for anything but large audiences.
• Programme holdout: customers excluded from an entire journey, such as onboarding or winback, for its full duration. Answers whether the programme works at all.
• Global holdout: a small share of the base excluded from all marketing contact for a quarter or longer. Answers what the channel is worth, and is the only version that catches cannibalisation between programmes.

The global holdout is the one that changes budget conversations, and the one that meets most resistance. The usual objection is lost revenue from the excluded group, which is worth taking seriously and worth quantifying. At one or two per cent of the base, the cost is small and the answer is permanent.

Assign membership once, from a stable random key on the customer record, and keep the assignment for the whole period. Reshuffling per campaign turns a clean experiment into noise.

Size it before you run it, not after

The most common way a holdout fails is quietly. Somebody excludes one per cent of a mid-sized audience, the result comes back with no clear difference, and the programme is declared unproven when the test never had the power to detect anything.

Work it out in advance, in this order:

1. Pick the metric. Revenue per recipient and purchase rate are the usual pair, and you choose one as the primary.
2. State the smallest effect that would change a decision. Not the effect you hope for, the one that would make you act.
3. Take the baseline rate and its variability from your own last ninety days.
4. Compute the sample size needed per group for that effect at your chosen confidence level, and check it against the audience you actually have.
5. If the audience is too small, either accept a longer run, accept a larger detectable effect, or move the holdout up to programme level.

Revenue metrics need more sample than rate metrics, because a handful of large orders can move an average on their own. If revenue per recipient is the primary metric, consider capping outlier order values and saying so before the test starts.

Write the analysis plan down before the first send. Metric, window, group sizes, stopping date, and what you will conclude from each outcome. Any A/B test benefits from that discipline, and a holdout requires it, because the temptation to check early and stop on a good day is strongest when the stakes are highest.

The hard part is making the exclusion hold

A holdout is only valid if the excluded group is genuinely excluded. In practice leakage is constant, and it comes from ordinary operational behaviour rather than from mistakes.

Places we look first:

• Transactional and service messages that carry a promotional banner or a recommendation block.
• Paid social audiences built from a CRM export, which quietly re-target the holdout.
• Push and in-app messages controlled by a different team with their own audience tool.
• Manual sends built by pasting a list, bypassing the suppression rule entirely.
• Any journey where a marketer has exclusion rights and a deadline.

Enforce the holdout as a suppression at the send layer, applied to every channel, owned centrally. Then audit it: each month, count messages received by holdout members. The target is zero and the number is rarely zero the first time you check.

Decide in advance whether service messages are in or out, and document the choice. Both are defensible. What is not defensible is discovering halfway through that nobody knew which it was.

Read it over a long enough window

A holdout read after two weeks tends to overstate the effect. Suppressing marketing does not remove demand, it delays it, and some of the difference you see early closes as the control group buys anyway.

Run for at least one full purchase cycle, and longer where the cycle is long. For a global holdout, a quarter is the sensible minimum, and a year is what you need before making a claim about customer lifetime value.

Report the confidence interval, not just the point estimate. A stated range is more useful to a finance team than a single number, and quoting statistical significance without the interval invites a precision the data does not support.

Then hold yourself to the result. Suppose the winback programme shows no incremental effect over a full cycle. The finding is that it does not work in its current form, and the next step is to change it or to stop it. That is the kind of evidence worth carrying into the year-end review, where budget for the following year is decided.

The uncomfortable part is what happens when a holdout says a favourite programme adds nothing. It happens often enough that you should agree, before you start, who gets to act on the answer. The measurement work sits in our optimisation practice, but the decision does not.