A/B vs. Multivariate Testing: Which One Is Better?

Posted by Julia Cardis, Senior Manager, Conversion Optimization

When it comes to testing, which method is better: A/B or Multivariate?  The optimizer in me wants to answer: “I don’t care—just test!”  But there are pros and cons with each approach—and ultimately, it depends on what your testing objectives are, and what you’re actually testing.

A Quick Explanation of Testing Techniques

A/B testing is the more straight-forward of the two testing approaches.  There is a champion and a challenger.  The winner is determined by performance of a single action—usually purchase or form submission.  But other actions, like entry rate to the site or use of a store locator, can also be used to determine the “winner.”  While  conducting A/B tests among existing pages to see which performs better (home page vs. category page vs. deep-link page) is a good starting point, Performics recommends testing custom landing pages for paid media traffic sources. (See our blog posting, Custom Landing Pages Enhance Marketing Effectiveness).

In multivariate testing, different elements on a page are tested (headline, image, long form or short form, no navigation or full navigation, call to action 1 or call to action 2, 3, or 4.  And so on.)  The number and combination of elements will determine the number of page variations and duration of the test—the more elements, the more combinations and the longer the test duration.  When the winner is determined, however, you’ll know not only which page combination is the winner, but how the individual elements contributed to the outcome.  More complex (and therefore more resources are required to launch the test) than a simple A/B test, but it’s easier to identify the element or elements that contribute to improvements. 


If You’re Testing Landing Pages from Paid Media or Controlled Traffic Sources

For paid media optimization, the best testing method is almost always A/B.  First, you’ll have fewer page iterations to design, code and tag.  So you’ll get results more quickly and can move on to the next iteration of the winning page.  You can always prove hypotheses around what page type or concept is the most effective, and move on to testing with different segments, messaging and offers.  More importantly, because you’re spending money to get the traffic in the first place, the sooner you can improve performance, the faster you’ll see an ROI on your ad spend.

Another reason we like A/B testing for paid media is that when you see big improvements in a short time frame, people take notice.  Enthusiasm builds, and often having a few big “wins” in your testing efforts can shift focus and resources to doing more testing.

While many marketers start A/B testing among existing landing pages, Performics recommends testing custom landing pages for paid media.  First, different traffic sources behave differently so you need a landing experience that’s best suited for the media.  Second, existing pages in a site are often too cluttered or are designed to meet the needs of many different visitors.   A custom landing page allows the marketer to use landing pages as an extension of the ad, and to help move visitors to the most desired action by taking the most relevant path. (See our blog posting, Custom Landing Pages Enhance Marketing Effectiveness).

If You’re Testing High-Volume Pages or Paths

For a home page or other high-volume entry pages, sometimes multivariate testing is a better approach because you can validate design decisions, navigation schemes and other page elements that have to perform for the highest common denominator.

Other scenarios in which multivariate testing might be a better approach than A/B testing is on critical paths like a shopping cart or a form.  For e-commerce sites, since the shopping cart is a universal path, you can test big changes like action buttons, form fields, field pre-fill, process indicators, and cart update/refresh to see if making enhancements improves order completion rates.  Always tag and monitor as many steps and cart elements so you can quantify exactly where and when users abandon the process.

The biggest obstacle to multivariate testing is test planning and managing page iterations.  The more you’re testing, the more complex the process.  If you don’t have the luxury of an experience test manager, you may find that the documentation and analysis of test results takes more time and resources than the value of the insights gained from testing.

But Before You Test Anything . . .

Before you decide which testing method is right for you, you have to decide if you’re even ready to test.  Testing involves planning, developing strategies, design and copywriting, code and development, and tagging of test pages.  If you don’t have adequate resources to support testing, focus your attention on building the business case to invest in testing.  Then determine what testing platform and method is best suited to meet your objectives.  Performics’ Conversion Optimization experts can help if you’re just getting started with testing, or have started testing and aren’t satisfied with the outcome of your efforts.

Comments are closed.

Performics Newsletter