After you activate an A/B test broadcast, its page shows the test's status, its settings and how each variation is performing. When the wait time ends, mCommons picks a winner. What happens next depends on your settings:
- Split Recipients: every recipient already got a variation. The winner is recorded for reporting, and nothing more is sent.
- Test Group with Auto-send Winner: the winner goes to the remaining recipients as soon as the wait time ends.
- Test Group with Manual Send: you review the results and choose which variation to send to the remaining recipients.
Broadcast statuses during a test
The status label appears next to the broadcast name.
Split Recipients
| Status | What it means |
|---|---|
| SCHEDULED | The broadcast is waiting for its scheduled start time. |
| PREPARING | mCommons is splitting recipients across the variations. |
| SENDING | The variations are sending. |
| AWAITING RESULTS | The variations finished sending and the wait time is still running. |
| Sent | The wait time ended and a winner was recorded for reporting. |
Test Group
| Status | What it means |
|---|---|
| SCHEDULED | The broadcast is waiting for its scheduled start time. |
| PREPARING | mCommons is picking the test group and splitting it across the variations. |
| SENDING | The variations are sending to the test group. |
| AWAITING RESULTS | The test messages finished sending and the wait time is still running. |
| WINNER IDENTIFIED | The wait time ended and a winner was picked. With Manual Send, the broadcast stays here until you send a variation. |
| SENDING WINNER | The chosen variation is sending to the remaining recipients. |
| WINNER SENT | The final send is complete. |
A winner is always picked when the wait time ends. There's no minimum number of responses.
Important: You can't stop an A/B test broadcast once it starts sending. Stop is only available while the broadcast is Draft or Scheduled.
Note: While a test is running, the broadcast list can show "0 messages" in the Length column, and the Messages link can open the edit page instead of the results.
Read the results page
Open the broadcast to see its results. The page has two main sections: Broadcast Details and A/B Test Results.
Broadcast Details
This section shows the settings the test uses.
| Field | What it shows |
|---|---|
| Delivery Time | The status and send time, for example "This is Awaiting Results to send at Today at 8:02PM EDT". |
| Test Duration | The wait time. While the test runs, it also shows time elapsed and time remaining, for example "1 hour (00:01 hours elapsed, 00:58 hours remaining)". |
| Recipient Splitting Type | Split Recipients or Test Group. |
| Broadcast Recipients | The number of included groups. Click View Details to see the groups and the estimated recipients. |
| Delivery Status | The latest status with a timestamp, for example "As of Today at 8:08PM EDT: Awaiting Results". |
| Broadcast Type | A/B Test. |
| Winning Criteria | The metric used to pick the winner: Response Rate, Click-through Rate or Engagement Rate. |
| Test Type | Auto Send or Manual Send. Test Group only. |
| Tags | Tags on the broadcast. You can add tags here. |
A/B Test Results
For Test Group broadcasts, the header shows Auto Send Enabled or Manual Send Enabled, so you know whether the winner sends on its own or waits for you.
With Manual Send, once the results are ready, you see a green Test Completed banner naming the winner and an orange No message has been sent yet banner. Both appear in the screenshot under Send the winner (Manual Send).
The results table has one row per variation:
| Column | What it shows |
|---|---|
| Ranking | The variation's place, such as 1ST PLACE and 2ND PLACE while the test runs, or WINNER once a winner is picked. |
| Variation | The variation name. |
| Winning % | The variation's score on the winning metric, based on the number of people, not the number of replies. It's capped at 100.0%. For example, 1 message sent with 3 replies from that person shows 3 Responses and 100.0%. |
| Metrics | Sent, Clicks (only when there are clicks), Responses and Opt-outs. Responses counts every reply, including repeat replies from the same person and opt-outs, so it can be higher than the number of people who replied. |
| Recipient Size | How many recipients got this variation. Click the number to see the recipients. |
| Message Body | The variation's message. |
| REPORT | Opens that variation's report. Appears once the variation has finished sending. |
The table updates while the test runs. If variations are tied, the earlier variation ranks higher. For example, if both Variation A and Variation B have a 0.0% Response Rate, Variation A shows 1ST PLACE.
Below the table, the Conversation section shows a phone preview of each variation with its current ranking.
For Test Group broadcasts, after the final send, the table drops Ranking and Winning % and adds a row for the copy sent to the remaining recipients. To see the ranking as it was when the winner was picked, use the Winning Snapshot.
Note: A/B test results aren't included in CSV exports.
Send the winner (Manual Send)
With Manual Send, nothing goes to the remaining recipients until you choose a variation. You can send the winner or any other variation.
The SEND WINNER and SEND THIS VARIANT buttons appear only after all of these conditions are met:
- The broadcast uses Test Group with Manual Send.
- The wait time is over and the results are in.
- At least one recipient is left to receive the message.
- Nothing has been sent to the remaining recipients yet.
Until then, the buttons don't appear on the page.
Important: There's no deadline for Manual Send. If nobody sends a variation, the remaining recipients never get a message.
Send the winning variation
- Open the broadcast.
- In the A/B Test Results table, click SEND WINNER on the winning row.
- In the Send Winning Variant dialog, click Yes, send winner. To close without sending, click Cancel.
Send a different variation
- Open the broadcast.
- In the A/B Test Results table, click SEND THIS VARIANT on the row you want to send.
- In the Send Selected Variant dialog, click Yes, Send This Variant. To close without sending, click Cancel.
After you send
- The status changes to SENDING WINNER, then WINNER SENT.
- The send buttons disappear. You get one final send only.
- The copy sent to the remaining recipients appears in the table as "variation name - Winner Copy", for example "Variation A - Winner Copy".
- The results stay on the page for reporting.
Note: The WINNER label in the results table uses current results. If replies arrive after the wait time ends, the label can move to a different variation than the one recorded in the Winning Snapshot.
Auto-send Winner
Auto-send Winner is available for Test Group broadcasts only. When the wait time ends, mCommons sends the winner to the remaining recipients right away. You don't need to do anything.
Important: The winner send doesn't follow Quiet Hours. If the wait time ends during Quiet Hours, the winner still sends.
When the send finishes, the status shows WINNER SENT, and the table includes the "variation name - Winner Copy" row.
View the Winning Snapshot
The Winning Snapshot shows the results captured when the winner was chosen. It includes Ranking, Winning %, and failed and undeliverable counts. It doesn't change when new replies come in.
The snapshot is available after the final send completes. You need editor permissions to see it.
- Open the A/B test broadcast.
- Click ACTIONS.
- Click Winning Snapshot.
View a variation's report
Each variation has a REPORT link that opens a detailed report for that variation. The link appears once that variation finishes sending.
The variation report shows:
- Summary: how many messages were sent to how many recipients for this variation, for example "1 message was sent to 1 recipient for this variation". Click View new subscribers count or View cleaned profiles count to load those numbers.
- Graph: replies (green) and opt-outs (red) over time, with zoom and date range options. The graph appears once the variation has replies or opt-outs.
- Current Conversation: the variation's message, with how many messages were generated, failed and sent, and how many responses were received.
- Responses: each response type with its count and percentage.
- Calls to Action: for each shortened link in the message, the total and unique link clicks.
- RCS stats (RCS variations only), including SMS fallback and MMS fallback counts for recipients who got the fallback message instead of RCS.
Why the report's numbers can differ from the results page
The variation report and the A/B Test Results table count some things differently:
| A/B Test Results table | Variation report | |
|---|---|---|
| Response rate | Winning % counts each person once and leaves out opt-outs. It never goes above 100.0% | Response Rate counts every reply. For example, 3 replies from 1 recipient shows 300.0% |
| Link clicks | Clicks for this variation's message | Calls to Action shows clicks for the link across every message that uses it, including the other variations and other broadcasts. The clickthrough rate note says how many other messages use the link |
Use the A/B Test Results table to compare variations. It uses the same rules as the winner metric.
Tip: Use a different shortened link in each variation. If variations share a link, the Calls to Action numbers in each variation report include clicks from the other variations.