When you set up an A/B test broadcast, you choose one winner metric. mCommons uses it to compare your variations and pick the winner. This article explains what each metric counts, how it's calculated, and which one fits your goal.
You choose the metric on the Recipients step, under What metric should decide the winner? in the A/B Test Settings panel, using the Select Metric dropdown. To see a quick summary of the metrics while you work, click the here link in "Click here for metric info" next to the dropdown.
Overview
| Metric | What it answers | What it counts |
|---|---|---|
| Response Rate | How many people replied? | Unique people who sent a reply that isn't an opt-out |
| Click-through Rate | How many people clicked my link? | Unique people who clicked an mCommons shortened link |
| Engagement Rate | How many people did something? | Unique people who clicked a tracked link or sent a reply that isn't an opt-out |
All three rates share these rules:
- Based on messages sent. Each rate divides by the number of messages sent to that variation, not the number a carrier confirmed as delivered.
- Unique people. Each person counts once per variation, however many times they reply or click.
- One decimal place. Results show rates rounded to one decimal place, for example 9.8%.
- Measured once. mCommons measures the rates when the wait time ends. Replies and clicks that arrive later don't change the winner.
Response Rate
Response Rate is the percentage of people who replied to your message, not counting opt-outs.
Response Rate = unique people who sent a reply that isn't an opt-out / messages sent x 100
What counts
| Counted | Examples |
|---|---|
| Positive replies | "YES", "Sure", "Count me in" |
| Negative replies | "NO", "Not interested", "Can't make it" |
| Questions | "What time?", "Where is it?" |
| Feedback | "Great message!", "This is spam" |
| Other replies | "Hello", "??", "K", or an emoji |
Opt-outs aren't counted. An opt-out is a reply that uses one of your account's opt-out keywords, such as STOP.
Example: basic calculation
You send a message to 10,000 people asking them to reply with their zip code.
| What happened | Count |
|---|---|
| Messages sent | 10,000 |
| People who replied with a zip code | 950 |
| People who replied with a question | 30 |
| People who replied STOP | 20 |
| People counted (zip codes plus questions) | 980 |
Response Rate = 980 / 10,000 x 100 = 9.8%
The 20 people who replied STOP aren't part of the rate.
Example: comparing two variations
- Variation A: "Reply YES if you support clean energy." Sent to 5,000 people. 600 replied, including 50 who replied STOP. Response Rate: 550 / 5,000 x 100 = 11.0%.
- Variation B: "What's your #1 concern about climate change? Reply now." Sent to 5,000 people. 750 replied, including 30 who replied STOP. Response Rate: 720 / 5,000 x 100 = 14.4%.
Variation B wins. Its open-ended question drew more replies.
When to use Response Rate
- Surveys, polls and data collection by reply
- RSVPs and confirmations by reply
- Testing which wording gets more people to reply
Click-through Rate
Click-through Rate is the percentage of people who clicked a tracked link in your message. Use it when your goal is to drive people to a website, donation page, petition or event registration.
Click-through Rate = unique people who clicked an mCommons shortened link / messages sent x 100
What counts
| Counted | Not counted |
|---|---|
| A click on an mCommons shortened (tracked) link | Clicks on links that mCommons didn't shorten |
| Clicks from any device | Extra clicks from the same person (each person counts once) |
Example: basic calculation
You send a donation appeal with a link to your giving page.
| What happened | Count |
|---|---|
| Messages sent | 20,000 |
| People who clicked the link | 1,400 |
Click-through Rate = 1,400 / 20,000 x 100 = 7.0%
Example: testing link placement
- Variation A (link first): "Donate now: [LINK]. Help us reach our $50K goal by midnight!" Sent to 10,000 people. 850 clicked. Click-through Rate: 8.5%.
- Variation B (link last): "Help us reach our $50K goal by midnight! Donate now: [LINK]" Sent to 10,000 people. 620 clicked. Click-through Rate: 6.2%.
Variation A wins.
When Click-through Rate is available
Click-through Rate is available in the Select Metric dropdown only when both of these apply:
- Every variation includes an mCommons shortened link.
- No variation is an RCS message.
If either condition isn't met, the option is unavailable.
Tip: Use a different shortened link in each variation. If variations share a link, the Calls to Action section of each variation report shows clicks from every message that uses the link, not only that variation.
When to use Click-through Rate
- Donation campaigns with a giving link
- Petitions and advocacy actions
- Event registrations
Engagement Rate
Engagement Rate is the percentage of people who did something: clicked a tracked link, sent a reply that isn't an opt-out, or both.
Engagement Rate = unique people who clicked a tracked link or sent a reply that isn't an opt-out / messages sent x 100
Someone who clicks and also replies counts once. That means Engagement Rate isn't clicks plus replies added together. When some people do both, it's lower than that sum.
Example: link and reply
You send a message with a donation link and ask people to reply with questions.
| What happened | Count |
|---|---|
| Messages sent | 20,000 |
| People who clicked the link | 1,500 |
| People who replied | 800 |
| People who replied STOP | 40 |
| People who sent a reply that isn't an opt-out | 760 |
| People who clicked and also sent a reply that isn't an opt-out | 300 |
Engagement Rate = (1,500 + 760 - 300) / 20,000 x 100 = 1,960 / 20,000 x 100 = 9.8%
You subtract the 300 people who did both, so each of them counts once.
For the same message, the other two metrics are:
| Metric | Calculation | Result |
|---|---|---|
| Response Rate | 760 / 20,000 x 100 | 3.8% |
| Click-through Rate | 1,500 / 20,000 x 100 | 7.5% |
| Engagement Rate | 1,960 / 20,000 x 100 | 9.8% |
Response Rate misses the people who only clicked. Engagement Rate includes them.
Example: link-only message
Your message has a link and doesn't ask for replies. Some people reply anyway.
| What happened | Count |
|---|---|
| Messages sent | 15,000 |
| People who clicked the link | 1,125 |
| People who sent a reply that isn't an opt-out | 45 |
| People who did both | 15 |
Engagement Rate = (1,125 + 45 - 15) / 15,000 x 100 = 1,155 / 15,000 x 100 = 7.7%
Example: reply-only message
Your message asks for replies and has no link.
| What happened | Count |
|---|---|
| Messages sent | 8,000 |
| People who clicked a link | 0 (no link) |
| People who replied | 1,200 |
| People who replied STOP | 40 |
Engagement Rate = 1,160 / 8,000 x 100 = 14.5%
With no link, Engagement Rate and Response Rate are the same.
When to use Engagement Rate
- Messages that include both a link and a request to reply
- Getting one number for overall engagement
- Comparing variations that use different calls to action
Side-by-side comparison
| Response Rate | Click-through Rate | Engagement Rate | |
|---|---|---|---|
| Measures | Replies | Link clicks | Clicks or replies |
| Includes link clicks? | No | Yes | Yes |
| Includes replies? | Yes | No | Yes |
| Includes opt-outs? | No | Not applicable | No |
| Needs a link? | No | Yes, in every variation | No (counts clicks if there's a link) |
| Available for RCS? | Yes | No | Yes |
Which metric should I use?
| Your goal | Metric | Why |
|---|---|---|
| Drive donations through a link | Click-through Rate | You care about clicks to your giving page |
| Get petition signatures through a link | Click-through Rate | You care about clicks to your petition |
| Collect survey answers by reply | Response Rate | You care about how many people replied |
| Collect RSVPs by reply | Response Rate | You care about how many people replied |
| Ask for a click and a reply in the same message | Engagement Rate | It counts both kinds of action |
| Measure overall engagement | Engagement Rate | It gives one number for everyone who acted |
How the winner is chosen
When the wait time ends, mCommons compares the variations on the metric you selected:
- The variation with the highest rate wins.
- If two or more variations tie, the earlier variation wins (A before B, B before C).
- There's no minimum number of replies or clicks, and no statistical significance check. mCommons picks a winner even if every variation scores 0%.
Tip: Use a test group large enough to produce real differences between variations. With a very small test group, a handful of replies can decide the winner.
RCS variations
- Replies to RCS messages count toward Response Rate and Engagement Rate.
- If an RCS message can't be delivered and mCommons sends the fallback message instead, replies to the fallback count toward the RCS variation.
- Click-through Rate isn't available when any variation is RCS.
Frequently asked questions
What's the difference between Response Rate and Engagement Rate?
Response Rate counts only replies. Engagement Rate counts replies and link clicks. Neither counts opt-outs. For example, 100 people reply (10 of them with STOP), 50 people click the link, and 20 of the people who clicked also sent a reply that isn't an opt-out. Response Rate counts 90 people. Engagement Rate counts 50 + 90 - 20 = 120 people.
If someone clicks and replies, do they count twice in Engagement Rate?
No. Each person counts once.
Do opt-outs count toward any winner metric?
No. Replies that use your account's opt-out keywords, such as STOP, aren't counted in Response Rate or Engagement Rate. Opt-outs appear in their own column in the test results, so you can still see them.
Why doesn't the Responses number match the Response Rate?
The Responses number in the test results counts every reply, including several replies from the same person and opt-outs. Response Rate counts each person once and leaves out opt-outs. For example, if 1 message was sent and that person replied 3 times, the results show 3 Responses and a Response Rate of 100.0%.
What counts as an opt-out?
A reply that uses one of your account's opt-out keywords, such as STOP.
What if someone clicks a link several times?
They count as one click.
Do replies that arrive after the wait time count?
Not toward the winner. mCommons measures the rates once, when the wait time ends.
Tips to improve your results
Response Rate
- Ask one clear, simple question.
- Make replying easy ("Reply YES or NO").
- Personalize the message, for example with a first name.
- Send at a time your audience is likely to read it.
Click-through Rate
- Place the link early in the message.
- Use a clear call to action ("Tap here to sign").
- Add urgency when it's real ("24 hours left").
- Test link placement and wording.
Engagement Rate
- Include both a link and a reply option when it fits your goal.
- Keep content relevant to the audience you're sending to.
- Avoid sending too often, which leads to more opt-outs.
Glossary
| Term | Definition |
|---|---|
| Sent | Messages mCommons sent to a variation's recipients. All three rates use this number, not carrier-confirmed deliveries. |
| Opt-out | A reply that uses one of your account's opt-out keywords, such as STOP. |
| Unique click | One click per person, however many times they click. |
| Tracked link | A link shortened through mCommons, so clicks can be counted. |
| Wait time | How long mCommons waits after the test messages send before it measures results and picks a winner. |