Flows

Split testing questions and offers

Sometimes you want to test different versions of the same question, offer, or form to see which performs better. Maybe a different headline gets more responses, or a different CTA drives more clicks. RightMessage lets you run split tests directly on your existing resources without duplicating anything.

This is different from A/B testing your website personalization, which measures personalized vs. default content across your site. Split tests here are about comparing variants of a specific question, offer, or form within your Flows.

How split tests work

A split test creates variant overrides on an existing question, offer, or form. RightMessage randomly assigns visitors to either the original (control) or one of your variants, then tracks how each version performs.

You can create multiple variants, not just one. For example, you might test three different headline approaches against the original to find what resonates best.

What you can test

Questions

  • Question headline text

  • Description text

  • Button text

  • Answer option labels and images

Offers and forms

  • Title

  • Description

  • Button text

  • Image

Setting up a split test

  1. Go to your Flow and find the question, offer, or form you want to test.

  2. Click the split test icon on the node.

  3. Create one or more variants. Each variant overrides specific parts of the original, so you only change what you're testing.

  4. Set the traffic split. By default, traffic is distributed evenly across all variants including the control.

  5. Click Start Test to begin collecting data.

  6. Publish your changes to make the test live.

Strategy tip: Test one thing per variant. If you change the headline and the button text at the same time, you won't know which change made the difference. Create separate variants for each change you want to measure.

Reading your results

Once a test is running, you'll see a split test card with live metrics.

High-level stats:

  • Unique visitors: Total exposures across all variants

  • Conversions: Total conversions with overall conversion rate

  • Best performer: Which variant is currently winning, with the improvement percentage

Per-variant breakdown: For each variant (including the control), you'll see:

  • Conversion rate

  • Improvement vs. control (shown as an up/down/flat arrow with percentage)

  • Raw counts (conversions out of exposures)

Statistical significance: RightMessage calculates confidence automatically:

  • Confidence level as a percentage (green when it reaches 95%+)

  • Estimated days to significance if you haven't reached it yet

  • Expand "Show details" for deeper stats: observed lift, p-value, 95% confidence interval, and a plain-English test recommendation

You can also preview all variants side by side to see exactly what each visitor group is seeing.

Ending a test

When you're ready to declare a winner, click End Test. You'll see each variant's final performance and can choose which one to keep:

  • The best performer is marked as Recommended

  • If you pick a variant that's underperforming, you'll get a warning

When you declare a winner:

  1. The winning variant's content is automatically applied to the original question, offer, or form. No manual editing needed.

  2. A snapshot of the test results is saved so you can review it later.

  3. The test moves to "Completed" status.

Don't forget to publish after ending a test. The winning content is saved to your draft config, but it won't go live until you publish.

Reviewing past tests

Completed tests appear as collapsed cards showing the test name, completion date, duration, and whether the result was statistically significant. Expand any card to see the full final metrics and which variant won.

Tips for effective split tests

  • Only run one test per resource at a time. Multiple simultaneous tests on the same question or offer will conflict.

  • Distribute traffic evenly when you have multiple variants. Three variants plus a control means 25% each.

  • Try meaningfully different approaches, not just minor word swaps. Test a completely different framing or tone alongside small tweaks.

  • Give tests time. The card will tell you estimated days to significance if you haven't reached it yet.

Was this helpful?
Was this helpful?