We all want to do more email testing, right? But what does that look like? And how do we effectively measure the success or failure of tests within email?

Let’s start with the tactics we often like to use – and then identify metrics that can answer the questions.

Before you get started, regardless of whether or not your email tool offers A/B testing suggestions or chooses automated winners, I recommend that you know how to and feel comfortable doing your own math. More importantly, know how your email tool records and calculates its metrics. 

For example, here’s information from MailChimp, Luminate Online, Marketo, or Engaging Networks.

I want to run a Subject Line test.

What’s the driving force behind this test? Could it be (1) answering *compelling* content questions, (2) getting better visibility in the inbox, or (3) you were just told to run tests?

The key metric for subject line testing is the open rate. Still, before you begin, you’ll need to ensure that your email deliverability and inbox placement are in good places with the audience to whom you’re sending this test. Otherwise, you can incorrectly interpret the metrics that your email platform is showing you.

A key misstep in a subject line test is ignoring the preview text options. This section is critically important! It should be adjusted to reflect the differences in the subject lines themselves. Litmus is a solid resource for getting your preview text spot on.

Another important consideration with subject line tests is the audiences that respond or don’t. Often prospects and active donors react differently so running a subject line test may provide more value to the larger audience segment.

I want to test the layout of my email messages.

What’s the driving force behind this test? Have you been using the same layout for X years? Are there significant layout differences between your types of emails?

The content of these emails must be the same to be correctly measured. The key metrics for this type of test are click-thru and response rates, with a few caveats.

If the layouts are drastically different and you’re adding more calls to action, you will need to also consider the math! Consider a few things as you configure the HTML of the email like adding an additional tracking parameter that means something unique to each link in each variant, like &link=header-c vs &link=header-t, &link=image-c vs &link=hero-t. This will allow you to measure click-through on specific design element differences more accurately.

If the test layout contains more calls-to-action, the response rate became more important than the click-thru rate. Response rate can also help if you’re looking at email messages over an email series within a campaign, because sending one stand-alone email with different layouts may not provide enough information for firm decision-making. The performance difference could be the email content, the visuals, the timing, or other factors.