Why I don’t A/B split-test emails…

I was reading this blog post from Klaviyo on “How to build an email marketing strategy that elevates the customer experience”

They gave 9 steps – 9 useful advices for anyone to add to their email marketing strategy…

Step 8 caught my eye:

===

(8) A/B test email elements one at a time

===

That’s good advice for split-testing in general…

(Although, it should be common sense that to test something you can only do it by having one dependent and all other variables independent… but all right.)

But it doesn’t really matter for email.

First:

If you do emails the Email Storyselling way…

Then getting the sale is NOT what you optimize for – building the relationship is.

And once you have a strong relationship with the customer…

…he doesn’t care whether you did/didn’t include “RE:” in the Subject Line.

He’s opening, reading, clicking and buying from your emails because of YOU.

But secondly:

It doesn’t really make sense to split-test emails anyway.

Most platforms (and I say most only because I haven’t gone through them but I’m pretty sure it’s basically all of them) don’t have enough split-testing sophistication to make the test scientifically viable.

What ESPs do is just divide your list into two buckets…

…and call that an A/B split-test.

Scientifically?

That has no validity whatsoever.

There are two pieces of information that must be as close to 50-50 as possible in an email split test:

First: Age of the account

(How long the person’s been on your email list.)

Second: Email domain

(Whether the person uses Gmail, Yahoo, Outlook etc.)

Someone who just opted in yesterday…

…is much more likely to respond to an email than someone who opted in 2 years ago. 

You want equal division between the two splits so that you get similar groups of people on each list.

Likewise…

The email domain is just as important:

You don’t want one split test group to be predominantly Gmails and the other to be predominantly Yahoos… their tab systems differ, their spam thresholds are different, and they have different reputation scores.

On top of this all:

Most email A/B tests are worthless because most aren’t meaningful – or significant enough – to create changes of 50-100%.

But anyway…