A/B Split Testing vs. Multivariate: Pros & Cons
“For Conversion Optimization, should I run an A/B Test or a Multivariate Test?”
The short answer is: “Yes”!
But, the question for today is: Which of the two primary testing methods should you employ?
(By the way, some are of the opinion that Conversion Optimization can be done without Controlled testing. That’s misguided, but a topic for another post.)
If you’ve been following Conversion Optimization for any length of time, you may be wondering how A/B Testing and Multivariate Testing work together.
There is an ongoing debate between proponents of Multivariate (MVT) testing and A/B Testing (aka, Split Testing or A/B/n Testing, where the “n” stands for any number of variations in a test) over which method gets better results.
Some people, especially technology vendors, believe Conversion Optimization testing and Multivariate testing to be two ways of saying the same thing. In their view, the technology and statistics behind the tests are what provides the value, and therefore the more complex Multivariate testing should deliver the best results.
I disagree: Multivariate testing does play a role in your Conversion Optimization Strategy, but it shouldn’t play the leading part.
There are distinct advantages to MVT, namely the ability to:
- Easily isolate many small page elements and understand their individual effects on conversion rate
- Measure interaction effects between independent elements to find compound effects
- Follow a more conservative path of incremental conversion rate improvement
- Facilitate interesting statistical analysis of interaction effects
But Multivariate Testing also has downsides:
- MVT usually requires many more variable combinations to be run than A/B/n
- MVT requires more traffic to reach statistical significance than A/B/n
- Major layout changes are not possible
- All variations within each “Swapbox” area must make sense together, which restricts the marketers freedom to try new positioning approaches
- The restrictions of the test setup constrain marketing creativity
- MVT either gives approximated results (if using Taguchi or similar test design) or requires exponentially more traffic (if using Full Factorial) than A/B/n
A/B/n Testing is much less dependent on advanced technology. It often does not take full advantage of a testing tool’s capabilities and is therefore much less interesting for technology vendors.
After all, there may not be a need for high licensing fees on testing tools if you can get the same results without spending much on the technology.
Advantages of A/B/n Testing
- The Conversion Strategist is not constrained by which areas of the page to vary
- You can test more dramatic layout, design, page consolidation and value proposition variations
- Advanced analytics can be installed and evaluated for each variation (e.g. click heatmap, phone call tracking, analytics integration, etc.)
- Test rounds usually complete faster than with MVT
- Often, you can achieve more dramatic conversion rate lift results
- Individual elements and interaction effects can still be isolated for learning
Disadvantages of A/B/n Testing
- The test rounds must be planned carefully if a goal is to measure interaction effects between isolated elements
- Any Taguchi or Design of Experiment setups must be planned manually
The “Software Agnostic” Approach
We take a “software-agnostic” approach at WiderFunnel. We find we can get better online results for our clients by not constraining ourselves by the features of a particular testing tool. In fact, we are happy to use and recommend a variety of testing tools.
This approach gives us freedom to recommend the type of test plan that will get the best results.
My Recommendation: Emphasize A/B/n Testing
The comfort MVT gives is enticing. But I’m a businessperson, and I recognize the significant opportunity cost of running prolonged tests that often are unavoidable with Multivariate Testing.
At WiderFunnel, we run one Multivariate Test for every 8-10 A/B/n Test Rounds.
In most cases, our first Test Rounds on a web page or conversion funnel start with A/B/n Testing of the major elements (like Value Proposition emphasis; page layout; copy length or eyeflow manipulation). In a third or fourth Test Round we may want to investigate interaction effects using MVT, usually only on pages with more than 60-100K unique visitors.
For more reasons, you should read: “Why A/B Split Test?“
Start with a Conversion Optimization Strategy
Before we even start testing, though, we create a Conversion Optimization plan that prioritizes where to start testing and when to use A/B/n or MVT.
Investing a few days at the start of a project to identify and prioritize the real business objectives, investigate the Web Analytics findings, understand the Value Proposition and Personas, and get the right tools in place pays off every time.