Configure A/B test settings

A/B test settings control how mCommons picks a winning variation, how long it waits before picking one, and how your variations reach your audience. You set them on the Recipients step of an A/B test broadcast.

Where to find these settings

The A/B Test Settings panel appears on the Recipients step when the broadcast type is A/B Test. Add your groups first. The recipient counts, split values and test group size are all based on your estimated recipients, so they stay at 0 until you add groups.

The panel has four settings:

• What metric should decide the winner?
• How long should we wait before picking a winner?
• How do you want to run your test?
• What should happen after the test ends? (Test Group only)

Choose the winner metric

The winner metric decides which variation wins. The variation with the highest rate for this metric is the winner.

1. Under What metric should decide the winner?, open the dropdown.
2. Select one of these options:
   • Response Rate (default)
   • Click-through Rate
   • Engagement Rate

To see how each metric is calculated, click Click here for metric info or see Understand A/B test winner metrics.

Note: Click-through Rate is greyed out if any variation doesn't include an mCommons shortened link, or if any variation is an RCS message. Hover over the option to see the tooltip: "Some metrics may be unavailable because one or more variations do not support it." Adding links doesn't turn it back on while an RCS variation is part of the test.

Set the wait time

The wait time is how long mCommons collects results before it picks a winner. The timer starts when the test messages finish sending, not when the broadcast starts. Results are measured once, when the wait time ends.

1. Under How long should we wait before picking a winner?, select one of these options:
   • 1 hr. (default)
   • 6 hrs.
   • 12 hrs.
   • 18 hrs.
   • 24 hrs.
   • Custom
2. If you selected Custom, enter a whole number of hours from 1 to 24. Minutes aren't allowed.

Choose how to run the test

Under How do you want to run your test?, select Split Recipients or Test Group.

Split Recipients

Split Recipients is the default. The helper text reads: "Send each variant to all your estimated recipients in equal portions. No automatic winner selection."

Every recipient gets one of your variations, and all variations send at the start time. mCommons still identifies a winner after the wait time so you can see it in your results, but nothing else is sent. Use this option when you want to compare results yourself after the campaign.

The split starts out equal across your variations. To change it:

1. Drag the anchors on the distribution bar, or enter a value in the Recipients or Portion field for a variation. You can also use the up and down arrows next to each field.
2. Check the totals. Each variation needs at least 1% of recipients, and the portions always add up to 100%.
3. To go back to an equal split, click the reset icon (Reset to equal split) on the right side of the bar.

For example, with 629 recipients, you can send Variation A to 449 recipients (71.4%) and Variation B to 180 recipients (28.6%).

Test Group

Test Group sends your variations to a smaller test group first. After the wait time, the best-performing variation goes to everyone else, either automatically or when you send it. The helper text reads: "Send variations to a test group, then automatically deliver the top performer to the remaining audience."

Note: The helper text says "automatically deliver" even if you choose Manual Send. With Manual Send, the winner only goes out when you send it. See Choose what happens after the test ends.

To set the test group size:

1. Under How many recipients will participate in the test?, enter a number in the field or drag the slider.
2. Check the number to the right of the slider. This is the Remaining Recipients count: the recipients who aren't in the test group and who get the winning message.
3. Review the table below the slider. It shows how many test group recipients get each variation.

The test group size starts at about 10% of your recipients. The smallest test group is one recipient per variation (for example, 3 for three variations), and at least 1 recipient must remain outside the test group to get the winner.

The test group is always split equally across your variations, and you can't adjust the split. If the test group doesn't divide evenly, the extra recipient goes to the first variation.

The Portion column shows each variation's share of your whole audience, not of the test group.

For example, with 629 recipients and a test group of 259:

Choose what happens after the test ends

This setting appears only when you select Test Group. Under What should happen after the test ends?, select one of these options:

With Manual Send, nothing goes out when the test ends. You review the results on the broadcast page and send the winner when you're ready. See Monitor results and send the winner.

When you're done, click SAVE & CONTINUE to go to the Schedule step.

Read the recipient graphs

The Recipients step shows two donut charts that update as you add groups and change your A/B test settings. Before you add groups, the charts are greyed out and the center shows 0.

Quiet Hours and the winner send

When you select Test Group and Auto-send Winner, this warning appears: "This send does not adhere to Quiet Hours, so the winning message will automatically send to the remaining recipients as soon as the test window ends." Split Recipients shows no warning on this step.

Here's how Quiet Hours apply to A/B test broadcasts:

• If your account has Quiet Hours turned on, you can't schedule the start time inside them.
• Once sending starts, A/B test broadcasts don't pause for Quiet Hours.
• With Test Group, the winner sends as soon as the wait time ends (Auto-send Winner) or when you click to send it (Manual Send), even during Quiet Hours.

Tip: Pick a start time and wait time that end inside your sending hours, or choose Manual Send. For example, a test that starts at 6 PM with a 6-hour wait sends the winner around midnight or later.

Related topics

• A/B testing overview
• Understand A/B test winner metrics
• Create an A/B test broadcast
• Schedule and test an A/B test broadcast
• Monitor results and send the winner
• Understanding broadcast metrics