A/B testing lets you send two or three versions of a broadcast, called variations, and see which one performs better. mCommons measures each variation using the metric you choose, such as replies or link clicks, and picks a winner.
Use A/B testing to answer questions like:
- Does a question get more replies than a statement?
- Does a link near the start of the message get more clicks than one at the end?
- Does an RCS rich card perform better than an MMS with the same image?
Before you start
- A/B testing must be turned on for your account. If you don't see the Broadcast type option when you create a broadcast, contact your mobile strategist or Technical Support.
- A/B testing isn't available on non-passthrough accounts.
- You need at least one more recipient than you have variations.
How an A/B test works
- Create the broadcast and choose A/B Test as the broadcast type. See Create an A/B test broadcast.
- Write two or three variations. Each variation is a single message. Variations can use different message types and channels, for example SMS against RCS.
-
Choose your test settings on the Recipients step. See Configure A/B test settings.
- Winner metric: Response Rate, Click-through Rate or Engagement Rate. See Understand A/B test winner metrics.
- Wait time: how long to collect results before picking a winner, from 1 to 24 hours.
- How to run the test: Split Recipients or Test Group (see below).
- Schedule the broadcast and send a test message. See Schedule and test an A/B test broadcast.
- Watch the results and, if you chose Manual Send, send the winner. See Monitor results and send the winner.
Split Recipients or Test Group
| Split Recipients | Test Group | |
|---|---|---|
| Who gets a message | Your whole audience, divided across the variations | A smaller test group first, divided equally across the variations |
| How the audience is divided | You choose the percentage for each variation (equal by default) | Always equal. You choose the size of the test group |
| What happens after the wait time | A winner is identified for reporting. Nothing more is sent | The winning variation goes to everyone who wasn't in the test group, automatically or when you send it |
| Best for | Learning what works for future broadcasts | Sending the best-performing message to most of your audience |
How the winner is picked
When the wait time ends, mCommons calculates the winner metric for each variation and picks the highest.
- Results are measured once, when the wait time ends. Replies and clicks that arrive later don't change the winner.
- Ties go to the earlier variation. If Variation A and Variation B have the same rate, Variation A wins.
- There's no minimum number of responses and no statistical significance check. A winner is always picked, even if every variation scores 0%.
Tip: Small test groups can produce a winner by chance. Use a test group large enough that a few replies won't decide the result.
Quiet Hours
If your account uses Quiet Hours, you can't schedule an A/B test broadcast to start during them. Once sending begins, A/B test broadcasts don't pause for Quiet Hours.
With Test Group, the winning variation is sent later than the test:
- With Auto-send Winner, it sends as soon as the wait time ends, even during Quiet Hours.
- With Manual Send, it sends when you click SEND WINNER.
Tip: Choose a start time and wait time that end inside your normal sending hours, or use Manual Send.
Limits
| Limit | Value |
|---|---|
| Variations per broadcast | 2 to 3 |
| Messages per variation | 1 |
| Wait time | 1 to 24 hours (whole hours) |
| Scheduling options | Send Now or a future date and time. Throttled, local time, fast track and recurring schedules aren't available |
| Click-through Rate | Available only when every variation includes an mCommons shortened link and no variation is RCS |
| Changing the broadcast type | Not possible after you leave the Details step |