Send-time optimisation: signal or noise

A client turned on their platform's send-time feature in the spring and reported a double-digit lift in open rate within a month. The number was real, in the sense that the dashboard was reporting it accurately. The revenue line for the same campaigns had not moved at all.

That gap is the whole subject. Send-time optimisation is one of the easiest features in a marketing platform to switch on, one of the hardest to evaluate, and one of the most confidently claimed. Almost every vendor publishes an uplift figure for it, and almost none of those figures come from a design that could distinguish the feature from the way it was measured.

We are not against it. We are against paying attention to it before the things above it in the list have been fixed. What follows is how to find out whether yours is doing anything.

What the feature is actually doing

The label covers at least three different mechanisms, and knowing which one you have changes what you should expect.

The simplest builds a single best hour for the whole list from aggregate past engagement, then sends everything then. The second builds a per-recipient model from each person's own open and click history and staggers delivery accordingly. The third adds a fallback for people with no history, usually the list-level best hour or a random draw inside a window.

That fallback matters more than it sounds. In most lists a large share of recipients have too little recent engagement to model, so a meaningful part of your file is receiving the default rather than the optimised time. Any measured effect is diluted by that group, and any reported effect that excludes them is measuring a selected, more engaged population.

Ask your platform three questions before testing anything. What signal does the model use, what is the minimum history required, and what happens to people below that threshold. If the answers are vague, that is itself a finding.

Why the reported uplift is usually an artefact

Open rate is the wrong metric here, and for send-time testing it is wrong in a specific, directional way.

Mail clients that pre-fetch images register opens without a person doing anything, and the proportion of your file behind that behaviour is large. Worse, a staggered send spreads delivery across the day, which changes how many messages arrive near the top of an inbox at the moment someone is looking. That produces a genuine change in the measured open rate without any change in customer behaviour at all.

There are three other common problems with the way this gets tested:

• The control is a single batch at one fixed hour. If that hour was chosen badly, the test measures the badness of the control, not the quality of the model.
• Delivery spread is confounded with delivery time. Staggered sending also reduces the load spike on your sending infrastructure, which can improve throughput and inbox placement independently.
• The comparison runs across different campaigns. Optimised sends in one week are compared with fixed sends in another, so content, audience and seasonality all vary alongside the treatment.

Any one of those is enough to produce a confident double-digit figure from a feature that does nothing.

A design that can answer the question

The test is not complicated, it is just more disciplined than the one the platform offers.

1. Randomise at the person level inside a single campaign. Same creative, same audience, same day. Split the audience into arms before any model is applied.
2. Run three arms, not two. Model-chosen time, a fixed hour chosen as your current best guess, and a random hour drawn from the same window the model uses. The third arm is what separates the model's intelligence from the mere act of spreading delivery.
3. Pre-register the metric. Conversion or revenue per recipient, decided and written down before the test runs. Opens and clicks are diagnostics, not outcomes.
4. Run it across many campaigns, not one. Send time effects, if they exist, are small. A single campaign cannot carry the weight, and repeating the split across a quarter of sends gives you something readable.
5. Work out the sample size first. Decide the smallest effect that would change your decision, then calculate how many recipients you need to detect it. If the answer exceeds your list, you have learned that this test is not available to you, which is a useful thing to learn in an afternoon rather than a quarter.

Point five ends most of these projects honestly. Statistical significance at the revenue level needs far more volume than significance at the open-rate level. A list that can readily detect a two-point difference in opens may need many months of sends to detect a one per cent difference in revenue per recipient.

The same measurement logic applies as for any programme-level claim, and we set it out in holdouts that prove it. A send-time test is simply an A/B test with an unusually small expected effect and an unusually tempting vanity metric.

Where send time genuinely matters

None of the above means timing is irrelevant. It means the general-purpose model is a weak lever, while several specific timing decisions are strong ones.

Content with a deadline. A flash sale, an event reminder, a delivery cut-off. Here the right time is dictated by the content and no model should be allowed to move it.

Time zones. Sending at ten in the morning in the wrong hemisphere is a bigger error than any model can correct. If you operate across markets and send on a single clock, fix that before testing anything subtler.

Regulated quiet hours for messaging. For SMS and other messaging channels, permitted sending windows vary by market and are a compliance matter rather than an optimisation one. Timing here is a constraint, not a variable.

Operational fit. If your contact centre cannot handle the replies a campaign generates at seven in the evening, the best send time is the one your business can answer.

Those four decisions are worth more than any per-recipient model, and none of them require a test to justify.

Bandits answer a different question

Teams often reach for a multi-armed bandit at this point, on the grounds that it will find the best time automatically without the test design above.

It will, in the sense that it will shift traffic towards whichever arm is performing. That is genuinely well suited to recurring triggered messages with stable volume, where you want continuous allocation rather than a verdict. It is poorly suited to the question in this article, because an allocation that changes as it learns makes a clean readout of effect size very hard to recover afterwards.

Pick the tool by the question. If you want to run a channel efficiently, a bandit is reasonable. If you want to know whether a feature deserves to stay switched on, you need a fixed split and a pre-registered metric.

The uncomfortable part

Send-time optimisation has a low ceiling. Even where it works, the effect is small relative to targeting the right people with the right offer, and testing it consumes volume that could have been spent on a question with a larger answer.

So put it near the bottom of the test roadmap and be honest about that ordering. When we review a year of a programme's tests, as in last year's review, the recurring pattern is a great deal of effort spent on small levers because they were easy to switch on. Send time is the clearest example of the type.