How campaign and A/B test verdicts work
You'll learn when a campaign holdback or A/B test can reach a verdict, what each state means, and why changing report dates doesn't change it.
Use the verdict to see whether there's evidence that a change helped more visitors convert. The same rules judge the campaign holdback ("Campaign vs holdback") and every Original page and audience A/B test. A higher rate alone isn't a verdict, and changing report dates doesn't start a new test.
The rules in short
- Weekly checks. Results are checked every Friday. Each week's visitors get 7 days to convert before they count, and only visitors measured on time count.
- Earliest verdict. The first verdict can come at the week 2 check, about three weeks after the test starts. Every version also needs at least 20 visitors who converted and 20 who didn't.
- Every week counts the same. Each counted week has equal weight, however many visitors it had.
- A false winner stays unlikely. Checking every week doesn't raise the chance of a false winner. When versions truly perform the same, about 1 in 40 tests names a false winner. With more than one challenger, that chance is split between them, so each one needs a little more evidence.
- Last check. The last check is at week 26. A version with no verdict by then is inconclusive. Tests never end on their own: you choose when to end them.
RightMessage never shows p-values, "% confident", or countdowns on campaign tests. Goal value is shown next to results but never decides a winner. Revenue per visitor on the Results tab is descriptive too and never changes a verdict. Question, form, offer, and flow tests still use their own confidence display.
What belongs to one measurement
Each measurement period keeps the content, conversion measure, split, and related settings together. Publishing a change to what's being measured starts a new period, and the publish dialog lists it under Restarts measurement. Earlier results stay in the test's history.
The conversion measure defaults to Primary goal. With at least two attached goals, Judge this campaign on also offers Any goal, which counts a visitor once if they complete any attached goal. Primary goal uses only that goal for the verdict, while other goals give context.
What each state means
| State | What it means |
|---|---|
| Collecting evidence | No verdict yet. A bigger observed difference alone doesn't change that. Conversion Goals source rows still call this Too early to call. |
| Winner | The whole likely range is better than Control (for the holdback: the campaign beats your page as it is). |
| Loser, or Did worse in a test's versions table | The whole likely range is worse. |
| Inconclusive, or No clear winner in a test | No verdict by week 26. It doesn't prove the versions are the same. |
| Can't reach a verdict | More than four weekly checks were missed, so the test can't decide. End it and choose a version yourself, or restart it to measure again. |
| No verdict | A version in a test that ended before a verdict. |
| Paused (health issue) | A health check failed, so verdicts are paused. The campaign keeps running. |
| Not decision-grade | Some of the evidence can't be trusted, for example because a publish couldn't be confirmed live. |
| Not measurable | Nothing to compare: the holdback is off, or the campaign lacks a usable goal. |
Health checks
| Check | What you see | What to do |
|---|---|---|
| Split mismatch: visitors weren't split as configured | Paused (health issue) and "A/B test paused: visitors weren't split as expected" in Needs attention | Check that the snippet loads on every page view, that every version can appear on the tested pages, and that no page cache serves one version to everyone. |
| Contamination: more than 5% of visitors saw more than one version | Paused (health issue) and "A/B test paused: some visitors saw more than one version" | Same checks. Visitors seen in both groups are left out of the results. |
| Unconfirmed publish | Paused · publish unconfirmed on a test, or Not decision-grade on the holdback. "RightMessage couldn't confirm when your last publish went live, so this test's results are on hold until it can." Counts show "—". | Nothing. It isn't a health problem and raises no alert. Results resume once the publish is confirmed. |
A missed weekly check adds no evidence, and the test runs a week longer to make up for it: "1 weekly check was skipped because it ran late. The test runs 1 week longer to make up for it."
Likely range
The likely range is the set of differences from Control that still fit the results. It narrows with each weekly check. A version wins when its whole range is better than Control and loses when it's all worse. A range that crosses zero means there's no verdict yet.
How much traffic to hold back
New campaigns with audiences start with a 10% holdback. A campaign without audiences starts with it off. After a week of traffic, RightMessage recommends a share based on how many visitors see the campaign each day.
| Estimated daily visitors | Recommended holdback |
|---|---|
| At least 1,000 | 10% |
| At least 100 but fewer than 1,000 | 20% |
| Fewer than 100 | A rounded value between 30% and 50% |
How many visitors should be held back? accepts 1 through 100. A 100% holdback leaves no personalized group to compare. To stop holding visitors back, turn off the Holdback toggle and publish. Existing browser assignments stay in place, so changing the percentage doesn't immediately reshuffle returning visitors.
What lift compares
How lift is measured describes the reference used in the report.
- When the holdback is on, its conversion rate supplies the comparison once both groups have at least 10 converters. With fewer, the campaign shows no lift yet, rather than a number from a handful of conversions.
- If the holdback is off and the campaign is unchanged, a previous comparison can describe the rollout.
- Otherwise, the reference can be similar visitors on the same pages who never saw the campaign, matched by engagement and known-contact status, once there are at least 10 conversions to compare.
That last comparison is descriptive, not randomized evidence. A positive difference against it doesn't make a Winner, and it never counts toward the headline lift on the Campaigns page or the dashboard's Personalization lift, which average only holdback and previous-test lifts.
If you turn the holdback off without changing the campaign, it can keep its previous Winner or Loser. RightMessage isn't measuring it again, and later changes to content, rules, goals, or segments can make it no longer apply.
Report dates and missing data
The date selector changes report figures, not the verdict. Engagement describes behavior but doesn't decide anything. An unavailable report doesn't mean zero conversions.
Until you pick dates yourself, the campaign list and the campaign's Overview and Results cover the whole test, labeled such as "Whole test, since Oct 5", up to the last 90 days. After a publish restarts the holdback's measurement, that includes the earlier runs, while the verdict describes the current run. If an earlier run held back a different share, "Lift mixes runs with different holdback shares." appears and the lift has no better or worse color. See which dates the lift covers.
The debugger shows the personalized content without changing your browser's real holdback or test assignment. If a matching visitor sees something else, use personalization troubleshooting to check that browser.
Tests started under the earlier rules
Holdbacks and tests that started before these rules shipped keep the previous rules until their measurement restarts. Under those rules, each weekly group was ready 14 days after its week started, decisions began at the second check with a limit of 26 checks, and both groups needed 20 converters and 20 non-converters. Publishing a change that restarts measurement moves them to the rules above.
For statisticians
The verdict is a two-sided group-sequential test on counted weeks, with O'Brien–Fleming-type error spending (Lan–DeMets) at an overall error of 0.05, split equally across challengers (0.05 ÷ m for m challengers). Information is spread evenly over 26 weekly looks, and the first look that can decide is week 2.
Each week's cohort contributes a score for the ratio of conversion rates (Farrington–Manning). Counted weeks are combined with equal weight. The likely range is the set of lifts whose combined score stays inside the boundary, so it excludes zero exactly when a comparison is decided. Decisions are final once made.
Boundaries on the z scale:
| Counted week | One challenger | Two challengers | Three challengers |
|---|---|---|---|
| 2 | 7.07 | 8.08 | 8.63 |
| 6 | 4.10 | 4.67 | 4.99 |
| 13 | 2.87 | 3.25 | 3.45 |
| 26 | 2.18 | 2.43 | 2.56 |
A missed week (measured late or during an unconfirmed publish) adds no information and extends the test by a week. The fifth missed week ends every open comparison as Can't reach a verdict.
For setup, see compare personalization with a holdback and run an A/B test on a page or audience. To plan around your traffic, see read results and test with low traffic.