The decision log that stops two years of tests evaporating

A new lifecycle manager joined a client in October and asked a reasonable question in her second week. Do we put a discount in the welcome series or not?

Nobody could answer. The test had been run, everyone agreed on that much. It was roughly eighteen months ago, the person who ran it had left, the platform's reporting only retained a year of campaign data, and the deck was in somebody's personal drive. What survived was a sentence in a strategy document saying the welcome offer had been "validated", with no number, no date and no definition of what it had been validated against.

So the team ran it again. Two months of a programme's testing capacity spent rediscovering something the company already knew. This is the most common form of waste we see in mature CRM programmes, and it has nothing to do with test design.

A decision log is not a test archive

The distinction matters, because most teams already have something that looks like a log and it does not solve this problem.

A test archive is a folder of results. It is organised by campaign, it grows without limit, and nobody reads it because finding the relevant entry takes longer than rerunning the test.

A decision log is organised by question. One row per question the business has settled, holding what was decided, on what evidence, and when it expires. A test that produced no decision does not get a row of its own. A decision made without a test does, because those are exactly the ones that get relitigated.

The test is evidence. The decision is the asset. Keeping only the evidence is how you end up with a shelf full of results and a team that cannot say what it believes.

The fields that make an entry useful two years later

Keep it short enough that people actually fill it in. We use nine fields, and the discipline is that an entry with an empty field is not finished.

1. The question, in plain language. "Does a first-order discount in the welcome series increase ninety-day revenue per new subscriber?" Not "welcome series test v3".
2. The date the decision was taken, and the period the evidence covers. Seasonality is the most common reason an old result stops holding.
3. What was actually compared. Both variants described well enough to rebuild them, including the audience definition and any exclusions.
4. The population and the volume. How many people in each arm, and which audience they were drawn from.
5. The primary metric and its measurement window, chosen before the test ran. One metric, not five.
6. The result, as an effect size with an interval rather than a verdict. "Between minus one and plus four per cent, centred near one" is a far more useful record than "no significant difference".
7. The decision taken, which is not the same as the result. Sometimes you adopt a variant that did not win, for cost or brand reasons, and that reasoning needs to be written down or it will look like an error later.
8. Confidence, in one word. High, medium or low, judged by the person who ran it.
9. Where the raw data lives, as a query, a table name or an export path that will still resolve after the platform is replaced.

Nine fields take about fifteen minutes to fill in. The test took six weeks.

Write the entry before the test runs, not after

Open the row when you design the test. Question, comparison, population, primary metric and window all get written down before a single send goes out, and nothing in that set may be edited afterwards.

This single habit fixes several problems at once. It stops the metric being chosen after the results are in, which is the most common way an A/B test produces a conclusion it does not support. It forces an explicit decision rule: what result would make us change, and what result would make us leave things alone. And it surfaces the tests that are underpowered before you spend six weeks on them, because writing the expected volume next to the expected effect size makes the arithmetic unavoidable.

Agree the stopping rule in the same sitting. When the test will be read, and that it will not be read early because someone is impatient. Peeking at a running test and stopping when it looks good is how a programme accumulates results that never replicate, whatever the significance figure on the dashboard says.

Record the results that found nothing

Most tests do not move the metric. That is a normal property of a well-run programme, not a failure, and those entries are the ones with the highest long-term value.

A null result tells the next person that this question has been asked, at this volume, and did not produce an answer worth acting on. Without it, the same idea returns in the next planning cycle wearing a different name, and the team spends another six weeks on it.

Be careful about how you phrase it. "No difference" is almost never what you found. What you found is that any difference was smaller than the test could detect, which is a statement about your sample as much as about the variants. Recording the interval rather than the verdict keeps that honest, and lets someone later judge whether it is worth retesting at a larger volume.

The same applies to programme-level evidence. When you run a holdout group and find that a journey's incremental contribution is smaller than everyone assumed, that finding must survive the person who ran it. The incremental read on a programme is the most expensive evidence you will produce, and it is also the most likely to be quietly forgotten when it is unflattering.

Give every decision an expiry date

An entry without a review date becomes folklore. Somebody quotes it three years on, the underlying conditions have changed, and no one can remember how strong the evidence was.

Set the review date by how fast the thing decays. Our rough guidance:

• Creative and subject-line findings: six months. Audience habits and competitor behaviour move underneath them.
• Offer and discount decisions: one year, or sooner if margin structure changes.
• Timing and cadence: one year. Everything we said about send-time optimisation depends on behaviour that shifts with device mix and seasonality.
• Structural decisions such as journey architecture or contact policy: two years.

When a date passes, the entry is not deleted. It moves to a review list, and somebody decides whether to retest, to reaffirm without evidence, or to mark it unresolved. All three are legitimate outcomes. The only bad outcome is an entry nobody has looked at since it was written, still being cited as fact.

Keep it where the platform cannot take it away

Put the log outside the marketing platform. Platforms get replaced, reporting windows expire, and accounts get closed when a contract ends. A plain table in the company's own document system or warehouse will outlive all of it.

Give it one owner, and review it in the same meeting where next quarter's tests are planned. Reading the last six entries before proposing new ones takes ten minutes, and it is the single cheapest way to stop repeating work.

The hard part is not the format. It is that nobody is rewarded for writing down a null result, and everybody is busy in the week a test finishes. The only fix we have found is to make the entry part of the definition of done, so a test is not finished when the deck is presented, but when the row is complete.