A/B testing is how you stop guessing. Instead of arguing about which subject line or layout will work better, you send both to small parts of your list, see which does better, and send the winner to everyone else.

It’s simple in principle. The two things that trip most people up are testing on too few people, and measuring the wrong thing. Both are easy to fix once you know about them.

What A/B testing is

You create two (or more) versions of an email that differ in one way, like the subject line, the sender name or the call to action. Each version goes to a random sample of your list. You compare the results on the metric you care about, then send the better version to the rest.

The point isn’t just to win one send. It’s to learn what your readers respond to, so every future email is a little better.

How to run a test in 4 steps

If you’re not sure what to test first, here are 14 email marketing tests worth trying.

1. Choose one variable

Pick one thing to change. Common options:

  • Subject line
  • Sender name
  • Preview text
  • Headline or opening
  • Call to action
  • Layout or visual elements
  • Send time

What you can test depends on your email software, but most handle subject lines and content at least.

2. Create your versions

Version A is usually your normal email, and version B has the one change. A common setup is sending A to 20% of your list, B to another 20%, then sending the winner to the remaining 60%. Some platforms let you test more than two versions at once.

3. Run the test

Send both versions at the same time, so timing doesn’t skew the result. Give it long enough for people to actually read the email. A few hours is often enough for a daily newsletter, but a weekly one may need a day.

4. Analyze the results

Compare the versions on the metric you chose, and check the difference is big enough to be real. A statistical significance calculator helps here. If each version went to a few hundred people and one got a couple more clicks, that’s almost certainly noise, even if your ESP declares a winner.

Judge tests on clicks, not opens

This is the big one. Subject line tests used to be judged on open rate. Since Apple Mail Privacy Protection arrived in 2021, Apple Mail loads emails automatically for many users, which records an open whether anyone read it or not. On most lists, that means a large share of “opens” aren’t real, and an open-based test can easily pick the wrong winner.

So judge tests on clicks, or on whatever you actually care about, like signups or sales, even if it means waiting a bit longer for the result.

A few habits that help

  • Test one thing at a time. If you change the subject line and the layout together, you won’t know which made the difference.
  • Keep everything else the same. Same audience, same send time, same content apart from the variable.
  • Use a big enough sample. Small lists can still test, but they need bigger differences to mean anything, and it’s often better to repeat a test a few times.
  • Write down what you learn. A dozen small tests on the same list, recorded somewhere, are worth far more than any single result.

Final thoughts

A/B testing isn’t complicated, and it doesn’t need fancy tools. Test one thing at a time, on enough people, measured on clicks, and keep a record. Do that regularly and you’ll know your readers far better than any benchmark report could tell you.

References

  1. [1] statistical significance calculator — neilpatel.com