The other day I had an interesting convo with an e-com brand founder.
He wanted to set up an A/B test around his email welcome sequence.
Variation A to promote a first-purchase discount code.
Variation B to communicate his brand’s founding story.
“It’s not worth your time,” I told him.
He looked baffled, to say the least, so I explained further:
“You don’t have enough transactions to run email A/B tests.”
Peep Laja from ConversionXL says you need roughly 350-400 conversions per variation to do A/B testing.
That’s not a rule, but a ballpark.
Now, let’s do the math together:
This brand makes roughly $2.5M in annual revenue.
That’s $200k per month.
Their average order value (AOV) is $114.
That’s 1,754 transactions each month.
Let’s say you craft an amazing welcome sequence and make 10% of monthly revenue from it.
Still, that’s only 175 transactions.
You set up the A/B test.
A month passes, and:
Variation A converts 93 customers.
Variation B converts 82 customers.
What have you learned?
That’s right, nothing.
You would need to run the test for multiple months to make any significant conclusions.
And doing A/B testing isn’t free. Tests take time to set up. Most e-commerce brands need to wait ages to reach statistical significance.
For a brand trying to scale, the opportunity cost is NOT worth it.
You’re better off focusing on other things, like crafting better Meta ads.
Bring in more traffic, more sales, more volume, and then do those A/B tests.
Should you A/B test email campaigns?
The example I just gave was about an automated email sequence.
What about email campaigns?
Should you test things like subject lines, send time, and product categories?
Unless you have hundreds of thousands of subscribers on your email list, the answer is still no.
The average e-commerce email campaign conversion rate is 0.08% (source: Klaviyo).
Say you send a promotional email campaign to 28,930 subscribers.
You test two different send times, morning and evening.
If you apply the average conversion rate, that’s only 23 sales.
Even if variation A generates 20 conversions, that’s not enough of a sample size to attain true statistical significance.
Alexandra Greifeld from No Best Practices points out another flaw with email split tests.
Your list contains people with different purchase intents. The variant that converts a recurring customer may not work on a new email subscriber who hasn’t made a single purchase.
To counter that, you could segment your audience between buyers and non-buyers, but then your sample size gets even smaller.
3 email marketing A/B tests actually worth running
So far, I’ve sounded like I have a personal agenda against A/B testing. I don’t.
But I do have an agenda against obsessing over stuff that doesn’t matter.
Growth isn’t found in testing campaign send times, flow time delays, or emojis in subject lines.
The moment you realize that, and start focusing on things that actually move the needle (like crafting better email content), your business will grow a lot faster.
I’m not saying A/B testing is completely worthless.
It’s incredibly useful when you test the right things.
The problem is, marketers are scared to be wrong, so they run meaningless tests around two similar variations.
Seth Godin says:
“Forget A/B testing. Consider A/J testing.”
I think Seth is on to something here.
To make your split tests useful, you need to test radically different alternatives.
Here are a few email A/B tests worth running for e-commerce brands:
- Pop-up incentives
Do you actually need to offer a first-purchase discount, or could you grow your list with a more profitable incentive, like a giveaway, free shipping, etc?
- Plain-text vs design
Does your audience respond better to plain-text, story-based emails, or highly visual designs?
- Promotions
What kind of promotions get customers to buy from you again? Can you offer gifts, discounts, or something else to increase retention rates?
Make sure you run these tests long enough to reach statistical significance.
You can still optimize without doing email A/B tests
Marketing VP Amanda Natividad says split testing is procrastination disguised as optimization.
She’s right — testing does not equal optimization.
If your e-commerce business doesn’t have enough transactional volume, you should still do qualitative research:
- Collect first-party data
- Survey your customers
- Ask for feedback from your support team
That’s my two cents. Protect your time, because most A/B tests in email marketing aren’t worth it.
