A/B Split Testing vs. Multivariate: Pros & Cons
“For Conversion Optimization, should I run an A/B Test or a Multivariate Test?”
The short answer is: “Yes”!
But, the question for today is: Which of the two primary testing methods should you employ?
(By the way, some are of the opinion that Conversion Optimization can be done without Controlled testing. That’s misguided, but a topic for another post.)
If you’ve been following Conversion Optimization for any length of time, you may be wondering how A/B Testing and Multivariate Testing work together.
There is an ongoing debate between proponents of Multivariate (MVT) testing and A/B Testing (aka, Split Testing or A/B/n Testing, where the “n” stands for any number of variations in a test) over which method gets better results.
Multivariate Testing
Some people, especially technology vendors, believe Conversion Optimization testing and Multivariate testing to be two ways of saying the same thing. In their view, the technology and statistics behind the tests are what provides the value, and therefore the more complex Multivariate testing should deliver the best results.
I disagree: Multivariate testing does play a role in your Conversion Optimization Strategy, but it shouldn’t play the leading part.
There are distinct advantages to MVT, namely the ability to:
- Easily isolate many small page elements and understand their individual effects on conversion rate
- Measure interaction effects between independent elements to find compound effects
- Follow a more conservative path of incremental conversion rate improvement
- Facilitate interesting statistical analysis of interaction effects
But Multivariate Testing also has downsides:
- MVT usually requires many more variable combinations to be run than A/B/n
- MVT requires more traffic to reach statistical significance than A/B/n
- Major layout changes are not possible
- All variations within each “Swapbox” area must make sense together, which restricts the marketers freedom to try new positioning approaches
- The restrictions of the test setup constrain marketing creativity
- MVT either gives approximated results (if using Taguchi or similar test design) or requires exponentially more traffic (if using Full Factorial) than A/B/n
A/B/n Testing
A/B/n Testing is much less dependent on advanced technology. It often does not take full advantage of a testing tool’s capabilities and is therefore much less interesting for technology vendors.
After all, there may not be a need for high licensing fees on testing tools if you can get the same results without spending much on the technology.
Advantages of A/B/n Testing
- The Conversion Strategist is not constrained by which areas of the page to vary
- You can test more dramatic layout, design, page consolidation and value proposition variations
- Advanced analytics can be installed and evaluated for each variation (e.g. click heatmap, phone call tracking, analytics integration, etc.)
- Test rounds usually complete faster than with MVT
- Often, you can achieve more dramatic conversion rate lift results
- Individual elements and interaction effects can still be isolated for learning
Disadvantages of A/B/n Testing
- The test rounds must be planned carefully if a goal is to measure interaction effects between isolated elements
- Any Taguchi or Design of Experiment setups must be planned manually
The “Software Agnostic” Approach
We take a “software-agnostic” approach at WiderFunnel. We find we can get better online results for our clients by not constraining ourselves by the features of a particular testing tool. In fact, we are happy to use and recommend a variety of testing tools.
This approach gives us freedom to recommend the type of test plan that will get the best results.
My Recommendation: Emphasize A/B/n Testing
The comfort MVT gives is enticing. But I’m a businessperson, and I recognize the significant opportunity cost of running prolonged tests that often are unavoidable with Multivariate Testing.
At WiderFunnel, we run one Multivariate Test for every 8-10 A/B/n Test Rounds.
In most cases, our first Test Rounds on a web page or conversion funnel start with A/B/n Testing of the major elements (like Value Proposition emphasis; page layout; copy length or eyeflow manipulation). In a third or fourth Test Round we may want to investigate interaction effects using MVT, usually only on pages with more than 60-100K unique visitors.
Start with a Conversion Optimization Strategy
Before we even start testing, though, we create a Conversion Optimization plan that prioritizes where to start testing and when to use A/B/n or MVT.
Investing a few days at the start of a project to identify and prioritize the real business objectives, investigate the Web Analytics findings, understand the Value Proposition and Personas, and get the right tools in place pays off every time.
April 26th, 2011 at 10:23 pm
You seem to have neglected to mention that A/B tests are limited to single pages of a website. If you have 10,000 products, then that's 10,000 individual A/B tests (in Google website optimizer at least). A multivariate chagne to the "Add to cart" button that affects all those 10,000 pages is the better choice for that test.
April 27th, 2011 at 10:02 am
I don't think it was a case of neglect at all. This article is obviously intended for a single page experiment for conversion testing. Most shopping carts are just cookie cutters with an image and description as the only difference from one page to another. An experiment of 10,000 products to see what effect "Add to cart" has on conversion is a gross misappropriation of capital. This article describes the more typical conversion tracking experimental need of the marketing and advertising professional. My only question is why Taguchi? I understand it in principal, but the results could be more practical with Plakett Burman.
April 27th, 2011 at 10:09 am
Thanks for commenting, Tom, so that I can clarify. Actually, we routinely run A/B/n tests on site-wide category pages.
As an example, take a look at this test we ran for BabyAge.com on their product page template: http://www.widerfunnel.com/proof/case-studies/bab…
Our talented developers have created many technical extensions to Google Website Optimizer to allow template testing in virtually any CMS platform.
April 27th, 2011 at 10:27 am
Some of the most popular testing tools have fractional factorial multivariate experiment designs built in. One of the popular tools we often use is Adobe (formerly Omniture) Test & Target, which uses Taguchi.
Another example of this is the test we did for WineExpress where the landing page changed daily based on a template: http://www.widerfunnel.com/proof/case-studies/41-…
Of course, we also do lots of non-template A/B testing too. For example, for Hair Club PPC landing pages: http://www.widerfunnel.com/proof/case-studies/hai…
May 27th, 2011 at 2:48 pm
Sorry for this Beginner-Question: How long do you let your tests run? Do you just at that moment you see a statistically difference? Or do you decide before starting that the test will for example last one week or 200 conversions? Do you have any strategy for this point?
May 27th, 2011 at 3:15 pm
It's not a bad question at all, Gerd. The decision of when to declare a test winner can be nuanced, depending on the situation.
As a rule-of-thumb, though, we aim to run experiments until a statistical difference at the 95% confidence level is achieved. But we also run tests for at least 10 days over two weekends to level out any day-of-week seasonality. Depending on the External Urgency, we may also recommend re-running a test at a different time to understand any longer-term seasonality effects.
May 27th, 2011 at 3:41 pm
Very good aspect not only to look at the confidence level but also on for example covering more than two weekends or repeat the test a couple months/week again. Thanks for this points…
August 15th, 2011 at 3:01 am
[...] 7. A/B Split Testing vs. Multivariate: Pros & Cons. [...]
August 15th, 2011 at 4:51 am
[...] 7. A/B Split Testing vs. Multivariate: Pros & Cons. [...]
August 15th, 2011 at 8:16 am
[...] 7. A/B Split Testing vs. Multivariate: Pros & Cons. [...]
August 17th, 2011 at 12:17 am
[...] 7. A/B Split Testing vs. Multivariate: Pros & Cons. [...]
August 20th, 2011 at 6:05 am
[...] 7. A/B Split Testing vs. Multivariate: Pros & Cons. [...]
September 7th, 2011 at 11:35 am
I think AB/MV testing is usefull, but a bit hyped at the moment. If you test all things like this until detail, you are probably optimizing for a select group of customers. Although this group might add to your revenue on short terms, you might discourage others who might be interested in e.g. next years collection. Don't make yourself vulnerable excluding others by taking AB/MV testing too far.
—- http://kinderbeddengoed.com
October 2nd, 2011 at 11:51 pm
[...] 7. A/B Split Testing vs. Multivariate: Pros & Cons. [...]
December 7th, 2011 at 12:24 pm
[...] A/B Split Testing vs. Multivariate: Pros & Cons – There is an ongoing debate between proponents of Multivariate (MVT) testing and A/B Testing (aka, Split Testing or A/B/n Testing, where the “n” stands for any number of variations in a test) over which method gets better results. Which of the two primary testing methods should you employ? [...]
December 7th, 2011 at 10:23 pm
[...] A/B Split Testing vs. Multivariate: Pros Cons – There is an ongoing debate between proponents of Multivariate (MVT) testing and A/B Testing (aka, Split Testing or A/B/n Testing, where the “n” stands for any number of variations in a test) over which method gets better results. Which of the two primary testing methods should you employ? [...]