A testing cadence that survives a busy quarter

Almost every CRM team we meet intends to test more. Very few do, and the reason is consistent. Testing is the first thing dropped when the campaign calendar gets heavy, because it is the only item with no external deadline.

The fix is not more enthusiasm. It is a cadence: a fixed number of tests in flight, a fixed day for decisions, and a backlog that someone owns. Rhythm beats ambition here, because a small test running every week compounds and a big test planned every quarter rarely ships.

This post covers how we set that rhythm up, size tests honestly, and keep the results from evaporating the moment the person who ran them changes role.

Fix the rhythm before the roadmap

Start by deciding how many tests you can genuinely have live at once. For most teams sending a few campaigns a week, the answer is one or two. Anything more and the audiences start overlapping.

Then put two recurring slots in the calendar. A short weekly session to launch what is ready and check what is running. A longer monthly session to read completed tests and update the backlog.

Assign one owner for the backlog. Not a committee. The owner decides what goes next, and their job is to make sure the queue is never empty when the weekly slot arrives.

Keep the backlog visible and ranked. We rank on expected impact divided by effort, and we require a written hypothesis before an idea can enter the queue at all.

Size the test honestly, then set the stop rule

Most CRM tests are underpowered, which is why so many read as "no clear winner". Before launching, work out the sample you need for the effect size you care about.

A rough guide we use: detecting a one percentage point difference on a ten per cent click rate needs tens of thousands of recipients per cell. Detecting a twenty per cent difference in conversion on a small segment may be impossible at your volume, and that is useful to know before you spend two weeks on it.

Set the stop rule at launch, not during the test:

• The fixed sample size or the fixed end date, whichever you chose.
• The metric that decides the outcome, named exactly, with the measurement window.
• The guardrail metrics that would stop the test early, such as unsubscribe rate or complaint rate.

Then stop looking at it daily. Peeking at results and calling a winner when the line crosses is the most common way CRM teams convince themselves of things that are not true.

When a test does produce a winner, check that it holds against revenue rather than clicks. The reasoning behind that sits in our note on CRM to revenue measurement.

Test one variable, and make it a big one

Change one thing per test. If you change the subject line, the layout and the send time together, a win tells you nothing you can reuse.

More importantly, make the one thing substantial. Testing two near-identical subject lines wastes a slot and returns a result you cannot detect. Testing a discount-led message against a value-led message returns something you can build on.

A rough order of impact we work through with most programmes:

• Who receives it. Segment definition and eligibility rules move numbers more than copy does.
• When it arrives. Timing, frequency and the trigger delay on behavioural messages.
• What it offers. Incentive level, incentive type, and whether there is an incentive at all.
• What it says. Positioning and message angle, before subject line wording.
• How it looks. Layout, imagery and call-to-action treatment.

Most teams start at the bottom of that list because it is the easiest thing to change. Start at the top instead.

Keep a decision log

A test result that lives in a chat message has a shelf life of about six weeks. After that the team retests the same thing, usually without realising.

Keep one file with a row per completed test: hypothesis, variants, audience, sample size, result, the decision taken, and the date. Ten lines each, no slide decks.

Record the losers and the inconclusive tests too. Knowing that send-time optimisation made no measurable difference last spring is worth as much as knowing what worked, and it stops the idea coming back every planning cycle.

Review the log before each planning session. The question is simple: what have we already learned that this quarter's plan ignores?

Retire what has done its job

Tests are not permanent. Once a variant wins and you roll it out, close the test and move the slot to something new.

Some results do expire. Creative fatigue, seasonal effects and audience mix all shift, so anything that shaped a major programme decision is worth re-running roughly once a year.

Audit your live tests quarterly and kill the ones that have been running for months without reaching their sample size. They are occupying audience and attention for nothing.

If you want an outside team to run this rhythm and hold the decision log, that is a standing part of our optimisation work.

What to do this week

• Put a recurring thirty-minute testing slot in the calendar and name one person as backlog owner.
• Take the next test you had planned and calculate the sample size it needs before you build it.
• Start a decision log with the last five tests you ran, including the ones that were inconclusive.