Compare personalization with a holdback

Find out whether your personalized page helps more people convert than the original. A holdback keeps some visitors on the original page, so you can compare people eligible for the same campaign. Holdback groups require Pro.

For example, WorkForge (opens in a new tab) could compare agency-specific homepage copy with its original homepage and measure demo requests. Audience variants decide which content fits someone. Holdback decides whether they receive the campaign's changes at all.

You'll need editing and publishing access, RightMessage installed on your site, and a campaign with tested page changes. Campaign holdback is separate from question, form, and flow tests.

1. Choose what you'll measure

Attach a conversion goal so RightMessage can compare the action you care about, such as a demo request.

1. Open the campaign's Conversion Goals tab.
2. Turn on Tracking for your goal, or choose New goal to create and attach it.
3. If you track several goals, choose Make primary for the one you want to use to judge the campaign.
4. Under Judge this campaign on, keep Primary goal when that choice appears.

Decide what matters before starting. Choosing whichever goal looks strongest afterward changes the question you're testing. Any goal is also available when you want to count a visitor once for completing any attached goal.

2. Set the holdback

1. Open Split Tests in the campaign.
2. Turn on Run an A/B test.
3. Set How many visitors should see your default content? to the share you want to hold back.
4. Review What to expect for the approximate traffic in each group.

Enlarge

Campaign Split Tests settings with Run an A/B test enabled and 20 percent of visitors assigned default contentOriginalOriginal

New campaigns start with testing enabled and 10% holdback. RightMessage can recommend a higher share for lower-traffic campaigns so both groups collect evidence.

3. Publish the test

Review and publish the full project draft once your page changes and test settings are ready.

The debugger shows the personalized page without changing your browser's actual holdback assignment. Use it to check your content, but remember that an ordinary visit can still receive the original page.

4. Read the comparison

1. Open Results after the campaign is published and enabled.
2. Choose the dates you want to inspect.
3. Compare the personalized and holdback visitor counts, conversion rates, and goal conversions.
4. Expand Who it converts for to see the combinations of personalized content visitors matched.
5. Read the decision state before you change the campaign.

Enlarge

The first four campaign experiences with personalized and holdback conversion rates, visitor counts and liftOriginalOriginal

You'll see the comparison build as visitors arrive and convert. A large observed lift isn't automatically a Winner. Decisions use scheduled checks of published experiment versions, not the dates you select in the report.

Next, read what campaign decisions mean before deciding whether to keep the personalized page for everyone.