Campaign experiment decisions and measurement limits
You'll understand when a campaign has enough evidence for a decision, how holdback works, and why changing report dates doesn't change the verdict.
Use your campaign's decision state to see whether there's evidence that personalization helped more visitors convert. The report also shows observed performance, but a higher rate alone isn't a decision. Changing report dates doesn't create a new statistical test.
What belongs to one experiment
Each published experiment version keeps the personalized content, conversion measure, holdback allocation, and related settings together. Publishing changes to the content or measurement setup can start a new version. Data you've already collected stays with its version.
The conversion measure defaults to Primary goal. With at least two attached goals, Judge this campaign on also offers Any goal. Any goal counts a visitor once if they complete any attached goal. Primary goal uses only that goal for Winner or Loser, while other goals provide context.
Publishing a different conversion measure starts a new experiment version. Changing report dates doesn't change that measure or reset the experiment.
When RightMessage checks the evidence
RightMessage groups newly enrolled visitors by week and gives each visitor seven days to convert. Each weekly group, called a cohort, is ready for assessment 14 days after its enrollment week starts. That gives people who arrive late in the week time to convert.
Decisions begin at the second scheduled check, with a limit of 26 checks. Counting starts with the first cohort whose full lookback period is still available. Earlier, incomplete cohorts provide descriptive data only.
Both the personalized group and the holdback group need at least 20 converters and 20 non-converters. Those counts alone don't guarantee a decision. The scheduled checks also assess the evidence and the experiment's health.
This is different from the statistical display in resource A/B tests. Don't apply that display to a campaign chart.
What each decision means
| State | What it means for your campaign |
|---|---|
| Collecting evidence, sometimes shown as Too early to call | The experiment hasn't reached a decision. More observed lift alone doesn't change that. |
| Winner | The scheduled evidence checks support the personalized treatment for the published conversion measure. |
| Loser | The scheduled evidence checks support worse performance for the personalized treatment. |
| Inconclusive | The evidence hasn't established a winner or loser within the applicable decision policy. It doesn't prove the versions are identical. |
| Paused (health issue) | Unexpected allocation or visitors appearing in both groups has paused decisions. The campaign itself can keep running. |
| Not decision-grade | The available comparison doesn't qualify for a controlled experiment decision. |
| Not measurable | The current experiment lacks a measurable conversion or usable measurement setup. Check the campaign's goals and status. |
How much traffic to hold back
New campaigns start with testing enabled and 10% holdback. The recommended share depends on how many visitors are estimated to see the campaign each day.
| Estimated daily visitors | Recommended holdback |
|---|---|
| At least 1,000 | 10% |
| At least 100 but fewer than 1,000 | 20% |
| Fewer than 100 | A rounded value between 30% and 50% |
These recommendations help both groups collect evidence. They don't promise a finish date. What to expect can be unavailable or ask for a shorter range if your reporting range is too large.
The field accepts 0 through 100. An enabled test set to 0 currently uses the 10% fallback. A 100% holdback leaves no personalized group to compare. To stop assigning new visitors to holdback, turn off Run an A/B test and publish. Existing browser assignments stay in place, so changing the percentage doesn't immediately redistribute returning visitors.
What lift compares
How lift is measured describes the reference used in the report.
- When the holdback has conversions, its conversion rate supplies the comparison.
- If testing is off and the campaign is unchanged, a previous A/B comparison can describe the rollout.
- Otherwise, the reference can be similar visitors on the same pages who never saw the campaign, matched by engagement and known-contact status.
That last comparison is descriptive, not randomized evidence. A positive difference against it doesn't establish a Winner.
If you turn testing off without changing the campaign, it can keep the previous Winner or Loser decision. Without holdback, RightMessage isn't measuring that decision again. Later changes to content, rules, goals, segments, or phrases can make it inapplicable to the current campaign.
Report dates, excluded visitors, and missing data
The date selector changes report figures, not the scheduled decision. Engagement shows pageviews and pages per visitor. It doesn't decide conversion success. Visitors seen in both groups are excluded from Results and flagged in Activity.
A health pause doesn't stop the campaign from appearing. An unavailable report doesn't mean zero conversions. Check the displayed state before you change settings.
The debugger shows the personalized content without changing the browser's real holdback assignment. If a matching visitor sees the original page, holdback is one possible reason. Use personalization troubleshooting to check that browser rather than inferring its assignment from the report percentage.
For setup, see compare personalization with a holdback. Campaign settings covers goal warnings and activity details.