Example: A/B test a get-out-the-vote message

Get-out-the-vote (GOTV) messages have a hard deadline: once the polls close, there's no second chance. Field research has found that even a simple text reminder can raise turnout. A/B testing helps you find the wording that gets the most supporters to act, while there's still time to send it to everyone else.

This article walks through a GOTV test from start to finish, and lists other ideas worth testing. If you're new to A/B testing, start with the A/B testing overview.

Plan around the calendar

• Test early, apply late. Election Day leaves little room to experiment. Run your tests during voter registration pushes or early voting, when you have days to learn, and use the winning approach in your Election Day messages.
• On Election Day, keep the test short. Use a Test Group with a short wait time (1 or 2 hours) and Auto-send Winner, so the winning message reaches the rest of your list while the polls are open.
• Mind the clock. You can't schedule an A/B test broadcast to start during your account's Quiet Hours. Once sending starts, A/B test broadcasts don't pause for Quiet Hours, and an auto-sent winner goes out as soon as the wait time ends. Choose a start time and wait time that end during reasonable hours.
• Follow your compliance rules. Election messaging often has extra consent and disclosure requirements. Check with your compliance team before you launch.

Plan a good test

• Change one thing at a time. Keep the call to action, link and length the same, and change only the element you're testing.
• Write down what you expect. For example: "Asking supporters when they'll vote will get more clicks than a plain reminder."
• Use a big enough test group. The smaller the difference you want to detect, the more recipients you need. At an 8% click-through rate, reliably spotting a lift to 10% takes about 3,200 recipients per variation. mCommons always picks a winner and doesn't check whether the difference is statistically significant, so check your results with an online A/B test significance calculator before applying them to Election Day.
• Let the wait time finish. Early results swing a lot. Results are measured once, when the wait time ends, and deciding early makes it more likely you pick a winner that isn't really better.

Choose the right winner metric

Tip: mCommons polling locators let supporters text a keyword with their address to get their polling place back. If your message asks supporters to use a polling locator keyword, their replies count toward Response Rate.

Walkthrough: plain reminder vs. vote plan

Question: Does asking supporters to plan when they'll vote get more of them to look up their polling place?

Hypothesis: The vote-plan version will get a higher click-through rate. Voter turnout research has found that helping people plan when and how they'll vote increases turnout, more than a general reminder does.

1. Create the broadcast and variations

1. Create a broadcast and select A/B Test on the Details step. See Create an A/B test broadcast.

2. In Variation A, write the plain reminder:

   {{first_name | default: 'Friend'}}, Election Day is Tuesday, Nov 3. Polls are open 7 AM to 8 PM. Find your polling place: [shortened link A]

   Use your state's actual poll hours. Opening and closing times vary by state, and sometimes by county.

3. Duplicate Variation A and rewrite the opening as a plan prompt, keeping the rest the same:

   {{first_name | default: 'Friend'}}, what time will you vote on Tuesday, Nov 3? Polls are open 7 AM to 8 PM. Find your polling place: [shortened link B]

4. Rename the variations, for example Reminder and Vote plan.

2. Configure the test

This test runs during early voting, 5 days before Election Day, to a list of 60,000 supporters. See Configure A/B test settings.

3. Schedule it

Schedule the broadcast for 10:00 AM. The winner goes out around noon, well inside normal sending hours. See Schedule and test an A/B test broadcast.

Before you activate, use Send test with All to check both versions on your own phone.

4. Read the results

The vote-plan version wins and goes to the remaining 52,000 supporters. A gap this size (7.9% vs. 9.8%) is statistically significant with 4,000 recipients per variation. See Monitor results and send the winner.

What to do next:

• Check the Opt-outs count for each variation.
• Use the vote-plan framing in your Election Day messages.
• If you have time before Election Day, test the next idea on the list below.

More ideas to test

Each idea below changes one thing. Keep everything else the same.

Tips:

• For social-norm messages, stress that many people are voting. Research has found that messages about high turnout motivate voters more than messages warning that turnout is low. Only use claims that are accurate for your audience.
• If one variation asks for a reply and the other doesn't, Response Rate favors the one that asks. Use Engagement Rate to compare a reply ask against a link, since it counts both.
• The first_name field holds up to 9 characters, so longer names are cut off. Always set a default, such as 'Friend', for profiles without a name.
• For RCS variations, recipients whose phones can't receive RCS get an SMS or MMS version instead, and their results count toward the RCS variation. See Create an A/B test broadcast.

Related topics

• A/B testing overview
• Understand A/B test winner metrics
• Configure A/B test settings
• Example: A/B test a donation appeal