Home / A/B test subject lines
A/B Test Email Subject Lines
Compare two subject lines side by side before you send, work out how many recipients each version needs, and check whether the winner really won or just got lucky.
1. Compare two subject lines
Our spring sale starts today
{first_name}, 25% off running shoes ends Sunday
| Factor | A | B | Note |
|---|---|---|---|
| Length | 25/25 | 25/25 | Tie |
| Word count | 10/10 | 10/10 | Tie |
| Spam triggers | 20/20 | 20/20 | Tie |
| Power words | 10/15 | 10/15 | Tie |
| Formatting | 15/15 | 15/15 | Tie |
| Personal | 0/8 | 8/8 | B: Speaks directly to the reader. |
| Specificity | 0/7 | 7/7 | B: Has a number, question or how-to that sets an expectation. |
Version B scores 15 points higher. Fix the weaker one before spending half your list on it.
2. How many recipients per version?
Why sample size matters
With a few hundred recipients, a 2-point difference in open rate is often just noise. The calculator uses the standard two-proportion sample size formula: the smaller the lift you want to detect, the more recipients you need. Halving the lift roughly quadruples the sample.
If your list is too small for the lift you care about, test bigger differences (a question vs. a benefit, not one swapped word), or test the same idea across several sends and look at the pattern.
3. Did your winner really win?
Opens or clicks?
Apple Mail Privacy Protection counts automatic opens, which inflates open rates (see why your open rate is inflated). In a random A/B split the inflation hits both versions about equally, so opens still work for comparing subject lines, but the difference gets diluted. Where you can, judge the winner on clicks or conversions too.
Pick the winner only after the test has reached the sample size you planned. Checking every hour and stopping when one side looks ahead inflates false positives.
Frequently asked questions
How do you A/B test email subject lines?
Write two versions that differ in one clear way, send each to a random split of your list (or a test group before sending the winner to the rest), wait until you have enough recipients, then compare open or click rates and check the result is statistically significant.
How big should an email A/B test sample be?
It depends on your usual open rate and the smallest lift you care about. For a 25% open rate and a 10% relative lift, you need roughly 4,900 recipients per version at 95% confidence and 80% power. Use the calculator above for your numbers.
How long should I run a subject line A/B test?
Most opens happen within the first day, but wait at least several hours, ideally 24, before choosing a winner, and don't stop early just because one side is ahead.
What should I test in a subject line?
Big differences teach the most: a question vs. a statement, a number vs. no number, personalised vs. not, short vs. long, with or without an emoji. Swapping a single synonym rarely produces a measurable difference.