Text messages are short, so a single change can shift how many supporters click through to give. This article walks through a fundraising A/B test from start to finish, and lists other ideas worth testing.
If you're new to A/B testing, start with the A/B testing overview.
Plan a good test
A test only teaches you something if it's set up to answer one clear question.
Change one thing at a time. If Variation B has a new image, new wording and a new ask amount, you won't know which change made the difference. Keep everything else identical.
Write down what you expect before you send. For example: "Adding a photo of our volunteers will raise the click-through rate." Deciding this in advance keeps you from explaining away a result after the fact.
-
Use a big enough test group. The smaller the difference you want to detect, the more recipients you need. At a typical 5% click-through rate:
- To reliably spot a lift to 7%, plan on about 2,200 recipients per variation.
- To spot a smaller lift, to 6%, plan on about 8,000 per variation.
With only a few hundred recipients per variation, a handful of clicks can decide the winner, and that winner may not hold up with your full list. mCommons always picks a winner and doesn't check whether the difference is statistically significant, so paste your results into an online A/B test significance calculator before you rely on them. Keep in mind that a bigger test group leaves fewer recipients to receive the winner.
Let the full wait time run. Early results swing a lot, and a variation that leads in the first few minutes often falls behind as more people read the message. Deciding early makes it much more likely you'll pick a winner that isn't really better. mCommons measures results once, when the wait time ends. Don't send a variation manually just because it's ahead after 20 minutes.
Build learning over several broadcasts. A single A/B test runs for 24 hours at most. To confirm a result, repeat the same test on your next appeal. A pattern that holds across two or three broadcasts is one you can rely on.
Choose the right winner metric
For most donation appeals, the goal is a click to your giving page, so use Click-through Rate.
- Put an mCommons shortened link in every variation. Click-through Rate isn't available if any variation is missing one.
- Use a different shortened link in each variation, even if they go to the same donation page. Add a different source or tracking code to each destination URL so your donation platform can tell you which variation brought in each gift.
- Click-through Rate isn't available if any variation is RCS. For RCS tests, use Engagement Rate.
Note: A/B testing picks the winner based on clicks, not dollars raised. Check your donation platform for gifts and revenue by source code before you decide what really worked.
Walkthrough: SMS vs. MMS
Question: Does adding a photo increase clicks enough to be worth the higher cost of MMS?
Hypothesis: The MMS version with a photo will get a higher click-through rate.
1. Create the broadcast and variations
Create a broadcast and select A/B Test on the Details step. See Create an A/B test broadcast.
-
In Variation A, write the SMS version:
{{first_name | default: 'Friend'}}, every gift this week is matched up to $50,000. Give now and double your impact: [shortened link A]
Duplicate Variation A. In the copy, attach a photo from your program and replace the link with shortened link B. Leave the wording exactly the same.
Rename the variations so they're easy to tell apart in results, for example SMS text only and MMS with photo.
2. Configure the test
On the Recipients step, add your donor group. In this example, the group has 40,000 recipients. See Configure A/B test settings.
| Setting | Choice | Why |
|---|---|---|
| What metric should decide the winner? | Click-through Rate | The goal is a click to the giving page |
| How long should we wait before picking a winner? | 4 hrs. | Gives most readers time to open the message and click |
| How do you want to run your test? | Test Group | Tests on part of the list, then sends the winner to everyone else |
| Test group size | 8,000 | 4,000 recipients per variation. The remaining 32,000 get the winner |
| What should happen after the test ends? | Auto-send Winner | Sends the winner as soon as the wait time ends |
3. Schedule it
Schedule the broadcast for 10:00 AM. The test messages finish sending shortly after that, and the 4-hour wait ends in the early afternoon, so the winner reaches the rest of your list well inside normal sending hours. A/B test broadcasts don't pause for Quiet Hours once sending starts, so plan the start time and wait time together. See Schedule and test an A/B test broadcast.
Before you activate, use Send test with All to check both versions on your own phone.
4. Read the results
When the wait time ends, the results might look like this:
| Variation | Sent | Unique clickers | Click-through Rate |
|---|---|---|---|
| SMS text only | 4,000 | 204 | 5.1% |
| MMS with photo | 4,000 | 252 | 6.3% |
The MMS version wins and goes to the remaining 32,000 recipients. A gap this size (5.1% vs. 6.3%) is statistically significant with 4,000 recipients per variation. A smaller gap, such as 5.1% vs. 5.5%, wouldn't be, and could easily be chance. See Monitor results and send the winner.
What to do next:
- Check the Opt-outs count for each variation. A winner that also drives more opt-outs may not be worth it.
- Compare gifts and revenue by source code in your donation platform.
- Weigh the lift against the extra cost of MMS. In this example, the photo brought in 48 more clicks per 4,000 messages.
- Run the same test on your next appeal to confirm the result.
More ideas to test
Each idea below changes one thing. Keep everything else the same.
| What to test | Variation A | Variation B | Metric |
|---|---|---|---|
| Ask amount | "Give $10 today" | "Give $25 today" | Click-through Rate (then check average gift in your donation platform) |
| Personalized ask | A standard amount for everyone | An amount from a custom field, for example {{suggested_gift | default: '25'}}
|
Click-through Rate |
| Framing | Urgency: "We're $4,000 short of tonight's goal." | Vision: "Together, we can open three new shelters this year." | Click-through Rate |
| Match | No match mentioned | "Every gift is matched 1:1 until midnight" | Click-through Rate |
| Emphasis | "Your gift will be matched" | "Your gift will be MATCHED" | Click-through Rate |
| Line breaks | One paragraph | The ask on its own line | Click-through Rate |
| Who it's from | "From all of us at [Organization]" | "This is Maria, our Executive Director" | Click-through Rate |
| Format | MMS with a photo | RCS rich card with a Donate button | Engagement Rate (Click-through Rate isn't available with RCS) |
Tips:
- Every message comes from your short code, so "who it's from" means who the message says it's from, in the opening line or sign-off.
- Personalized ask amounts need a custom field that stores a suggested amount for each profile. Always set a default for profiles without one. See the Advanced Liquid article.
- Capital letters draw attention, but use them sparingly. All-caps text is slower to read, and a message that's mostly capitals can read as shouting. Capitalize one key word, not whole sentences.
- Fundraising research has found that mentioning a matching gift raises response rates, but larger matches (2:1 or 3:1) didn't do better than 1:1. If you have a match, test how you present it rather than the ratio.
- If a variation includes a reply option (for example, "Reply GIVE to donate"), use Response Rate or Engagement Rate, and make sure the other variation asks for the same action.