Some people are disappointed when they learn that their split test results are not 100% precise. I have to admit that I hesitate writing this article because I know it can create unnecessary disillusionment.
If you think statistics are infallible you’ll be surprised to learn that it’s not as straight-forward as you thought. To put statistical accuracy into perspective, when searching for the Higgs-Boson particle, CERN had 3 teams dedicated to poring over data for errors every day.
Scientific testing, however is still the best method for marketing optimization. Questions of accuracy and precision are common and important to understand.
What is exact, exactly?
When WiderFunnel delivers a statistically significant test result for our clients, we use a 95% statistical confidence level. That means you can be 95% confident in the result.
At that accuracy level, the result is expected to be consistent 19 times out of 20.
“Sure, the conversion rates are accurate to within a fraction of a percent, 19 times out of 20,” some say, “But I want it exact, dammit!”
Let’s take a step back and see what we’re talking about.
Why are you running conversion optimization tests?
You want to make the best decision you can in your marketing, right?
But, you can’t always make the best decision because you don’t have enough information. You’re expected to make the right call every time without the benefit of hindsight.
I empathize with your situation. That’s a lot of pressure and it’s understandable that making decisions sometimes requires a bottle of Pepto-Bismol.
That’s where testing comes in.
Testing offers a respite from decision-guessing pressure.
Instead of guessing, you can test to find out what works better, and the information you get from testing is statistically significant.
Tell me, how often do you get information that’s correct 19 times out of 20?
- Is your intuition that good?
- How about your focus group feedback?
- Can you judge the effectiveness of a marketing message that precisely, every time?
I’m being facetious here, but only a little. If your intuition is even correct 11 times out of 20, I have a strong recommendation for you- stop what you’re doing right now and go to Vegas.
In reality, many of us have very little information as accurate as a controlled test result.
What do conversion optimization test results mean?
Let’s say you get a test result showing a 16.5% conversion rate lift at a 95% confidence level. Great! You should be happy about that. That’s a significant result from a single test!
I’m going to share a little secret that may surprise you; the result doesn’t necessarily mean that you’ll get a 16.5% lift on an ongoing basis.
You may get more. Or less. But, there’s a very good chance (a statistically significant one, in fact) that you will see a conversion rate lift.
According to statistics, the exact amount of the conversion rate lift can’t be known with the amount of data available. For Challenger 1, we can be 95% confident that it will be in the range of between 6% and 28%.
“Wait a minute!” you say, “That’s a big range! Isn’t that inaccurate?”
No, this result is very accurate for the question we asked: “Will it lift conversion rates?”
The answer is, “Yes, it will (and I’m 95% confident).” As a business rule for decision-making, a 95% confidence level is a very high hurdle to clear.
If your question is, “by how much will it lift conversion rates?” then the answer requires a higher level of precision and more data.
If that’s your question, you have a cost-benefit business decision to make:
How much time is that extra precision worth?
You can always keep running the test longer, but there’s an Opportunity Cost. You could use that time and traffic to run another test and have a shot at even more conversion rate improvement.
Incidentally, Challenger 3 in this example is also likely to beat out the Control but it hasn’t proven itself yet. Its minimum value within the Conversion Rate Lift range is still negative, so we won’t call it a winner yet.
What is the ‘real’ conversion rate lift?
Unlike the Higgs-Boson particle, there is no ‘real’ conversion rate other than the conversion rate you track and observe. If you take away one thing from this article, it should be this: statistics is not attempting to reveal your ‘real’ conversion rate.
With more data, statistics will tell you that you can be more confident in the observed conversion rate. And while statistics concerns itself with probabilities and confidence, business is about making good decisions.
The most accurate conversion rate you have at any time is the one you’ve observed. In other words, the ‘real’ conversion rate lift is the observed conversion rate lift at this point: 16.5%.
Should you make decisions based on split test data?
For me personally, I would accept the accuracy of a split test any day of the week.
In the end, you can either take the winning result, which is likely to give a 16.5% (plus or minus) conversion rate lift, or keep the Control page, which probably wasn’t tested in the first place.
Which would you rather bank on?