Compare personalization with a holdback
You'll hold some visitors back from your campaign, then see whether the personalized page converts better than your page as it is.
Find out whether your personalized page helps more people convert than your page as it is. A holdback keeps some visitors out of the campaign, so you can compare people eligible for the same campaign. The holdback is on the Scale and Pro plans.
For example, WorkForge (opens in a new tab) could compare agency-specific homepage copy with its usual homepage and measure demo requests. Audience rules decide which content fits someone. The holdback decides whether they get the campaign's changes at all.
You'll need editing and publishing access, RightMessage installed on your site, and a campaign with page changes.
The holdback and A/B tests
The holdback measures the whole campaign. Each campaign has one. To compare versions of a page instead, run an A/B test, which needs Pro:
| If you'd like to… | Use |
|---|---|
| Know whether the campaign as a whole beats your page as it is | The campaign holdback (this article) |
| Compare versions of the page visitors see when no audience matches | An Original page test |
| Compare versions of one audience's changes | An audience test |
Campaigns don't need segments. A campaign with no audiences is a plain page A/B test of the Original page. See run an A/B test on a page or an audience. Question, form, offer, and flow tests are separate and available on every plan; see test questions, forms, and offers.
1. Choose what you'll measure
Attach a conversion goal so RightMessage can compare the action you care about, such as a demo request.
- Open the campaign's Conversion Goals tab.
- Turn on Tracking for your goal, or choose New goal to create and attach it.
- If you track several goals and want one to decide the campaign's verdict, choose Primary goal under Judge this campaign on.
- Choose Make primary for the goal you want to use.
Decide what matters before you start. Choosing whichever goal looks strongest afterward changes the question you're testing. Any goal counts a visitor once for completing any attached goal.
2. Set the holdback
- Open the campaign's Testing tab.
- In the Holdback card, turn on the toggle.
- Set How many visitors should be held back?. It accepts 1% to 100%.
- Check the split bar under the field for the approximate traffic in each group.
New campaigns with audiences start with the holdback on at 10%. Once the campaign has a week of traffic, RightMessage recommends a share, such as "We recommend 20% for about 300 visitors a day.", and tags the matching value Recommended. Until then it says "We'll suggest a share once this campaign has a week of traffic."
Held-back visitors see your page as it is. If the campaign also has an Original page test, held-back visitors skip the audiences but still see the Original page test, so the holdback measures what the audiences add.
To stop holding visitors back, turn off the toggle and publish. The card then shows Turn on holdback and the share you'd hold back. Existing browser assignments stay in place.
3. Publish
Review and publish the project draft once your page changes and holdback are ready.
The debugger shows the personalized page without changing your browser's real holdback assignment. Use it to check your content, but remember that an ordinary visit can still be held back.
4. Read the comparison
Open the Testing tab. The right side of the Holdback card is Campaign vs holdback:
- Personalized, Held back, and Difference, with each group's converters and visitors.
- A plain sentence about where the campaign stands.
- Likely range: the differences that still fit the results, on a bar from Campaign worse to Campaign better.
- Difference by week: a descriptive chart. Only the weekly checks can call a verdict.
The status tag shows the verdict: Collecting evidence, Winner, Loser, Inconclusive, Can't reach a verdict, Paused (health issue), Not decision-grade, Paused · campaign off, or Not measurable when the holdback is off. The Results tab adds goal comparisons, Who it converts for, and engagement for the dates you choose.
Until you pick other dates, Results covers the whole test, including runs from before a publish restarted the holdback's measurement. The label beside the lift names the dates, such as "Whole test, since Oct 5". See which dates the lift covers.
A large observed lift isn't automatically a Winner. Verdicts come from the weekly checks, not from the dates you pick in a report, and the earliest comes at the week 2 check. Read how campaign verdicts work before deciding to keep the personalized page for everyone.
5. See revenue per visitor
If you've connected Stripe, the Results tab also shows Revenue per visitor under the goal comparisons, for the dates you choose. It appears only while the holdback is on, so there's a held-back group to compare.
It shows what each group paid per visitor, such as "Personalized $96.29 · Held back $94.04", and the difference with its range, such as "Difference +$2.24 per visitor (range −$50.84 to +$55.33)". Until each group has 5 buyers and 30 visitors, it shows the counts so far instead, such as "3 and 2 buyers so far, out of 40 and 12 visitors."
Revenue comes from Stripe payments by identified visitors after they first saw the campaign. Visitors RightMessage never identified count as $0 in both groups, so the Identified line matters: "Identified: 88.8% of personalized and 90.8% of held-back visitors." When the two groups were identified at different rates, it warns you, because matched revenue can then differ for reasons other than buying. People who saw both versions aren't counted in either group.
Revenue per visitor is descriptive only. It never changes the campaign's verdict, which still comes from your goal and the weekly checks.