Testing: What do you really learn?

On the surface, it seems quite straightforward: test a subject line here and there, maybe a different creative, maybe different offers to different segments. In actuality, it is likely the most misunderstood practice in email marketing and the most poorly administered.

Here's my view of testing, how to break it down and how to develop a testing strategy.

Testing starts with developing a hypothesis and a log. Why? You can't optimize without a hypothesis of what you hope to achieve and a planned action behind it. Without storing a history of your tests or what worked and what didn't, you will likely recycle the same tests over and over again.

So, what are the things you can test, and why are they important?

Targeting – Which segments will perform best to existing promotions. Could be based on past purchase, customer state, value, product affinity, site behavior or simply cohorts that you've established. This is the best area to test cadence, pull out a few customer segments and hold out a group; you'll see incremental effects of frequency on different customer segments.

Promotion – This can be reward or incentive, but testing variances in promotions is probably the most typical method of testing today. 20%, 30% off, free shipping, multiple purchase discounts. Many test this in simple fashion A/B style, primarily due to the complexity this can cause on different value segments and cadence.

Design – Obviously this is talked about a lot in the email world. Short versions, long versions, use of product images, text links, layout... This type of testing is best to do in a multivariate scenario, as it will save you a lot of time during the production crunch. The difficulty is isolating it to a few key elements that you can take forward into all your email communication streams. If you test a promotional email, will the 'learnings' pass to your newsletter and triggered messaging? It depends on your hypothesis.

Subject Lines – As we all know, subject lines are the "context" for which the receiver decides to open the message (although the 'from' address/name is important to "scanners" as much as "poll position" is). What do you test? There are essentially three main elements you can test: Brand-specific subject lines, action-oriented subject lines and benefit-driven subject lines, which can be combined to test infinite combinations. Add personalization to this and placement (front, middle, end) and you get an interesting testing matrix. It's critical that you log these types of tests, as they are often done very adhoc, which means that at the end of a year, no one can tell you the top winning subject lines, tests or approaches.

Landing Pages – Don't forget the destination! Just because they clicked through doesn't mean they will do what you want. Landing page tests are best suited for multivariate testing as well. There are simply too many variables to test, from abandonment, form layout, sequencing of pages, and all the creative variables. Small changes can have dramatic impact on performance.

Testing isn't just about finding a winning solution, it's a process of trying to find the greatest variance in tactics and validating your hypothesis.

Most companies I know have tested A/B methods, some have tested multivariate, and very, very few even think about doing more sophisticated methods like Taguchi. Just remember, adding complexity to the test doesn't make it valuable, it just makes it a bit more efficient – so if you have limited bandwidth, it may make sense to do less frequent, but more in-depth, testing a few times a year.

As I've preached for years, we don't test unless there is a prescribed action associated with the winning hypothesis; this includes even simple things like subject lines. Ask yourself, if "Your bonus offer is ready to view" performs better than "Get your bonus offer today" as a subject line test, what action will you take in response? Does it mean your audience is averse to directive statements, and you'll sequence the next few messages with more passive language?

Testing can be fun, it can be a lot of work, but your job as a marketer is to find the most efficient methods to build ‘learnings' about your customers and their behaviours with email.

Article courtesy of http://publications.mediapost.com/