I Love Losing Test Results and You Should Too
Recently, I gave a keynote presentation at the European Conversion Summit in Germany, and I revealed the best-kept secret of conversion optimization: Not All Tests Win. Tweet this:
Sadly, it’s true.
At WiderFunnel, we often share case studies of winning test results publicly. Most often, they show spectacular winning results and marketing insights.
While many people are understandably reluctant to acknowledge that they don’t always produce winners, I’m happy to. I’d like to tell you why.
Conversion Optimization Tests Don’t Always Win
I admit it. There are times that I’ve allowed conversion optimization test results to ruin my day.
You see, at WiderFunnel, we monitor every test on a daily basis so that we can make tactical changes to maximize ROI.
When there are tests on the dashboard that don’t show strong winning results, it’s hard not to let them get me down. I want to win big every time!
But We Do Win Most of the Time
Fortunately, the awesome team at WiderFunnel has an enviable success rate for tests.
I’ve had many conversations with internal testing teams and other agencies. Most companies report that around a third of their conversion optimization tests, out of every three, produce winners.
An agency that runs tests regularly may get up to 50-60% wins, which is very good – about double the internal teams rate. With WiderFunnel’s system, we’re getting a much higher success rate. In our last quarter, 89% of tests planned by WiderFunnel produced a new champion page. Tweet this:
I should be thrilled. But… there’s still that 11 percent that didn’t deliver a new winner.
Take a look at these two test results as examples. Staring at these every morning, day after day… well, it gets old, fast.
Clearly a test with the big winner is more fun to see. But, an inconclusive test doesn’t annoy me quite as much as it used to.
Why I Love Inconclusive and Losing Test Results
Why I Love Inconclusive and Losing Conversion Optimization Test Results
Over the years, I’ve learned and re-learned that those losing and inconclusive tests hold valuable insights.
In fact, they can be so valuable that they should be called winners too.
We’ve seen over and over again that a commitment to continuous testing gets the best results. We often see the best results on the 2nd, 3rd and 4th tests on a page or template.
But, results don’t just come from repetition. Great results come from making insightful learning from each and every test. That’s important to understand: great results come from powerful insights.
Turn Inconclusive Tests into Winners
Where that inconclusive test I mentioned earlier would have ruined my day in years past, now our team looks at it as an opportunity.
We analyze the results of inconclusive tests to understand the differences and produce hypotheses that will get better results.
In the case of this client with an inconclusive test, WiderFunnel’s next test round produced a statistically significant 16.1% winning result.
Sometimes the follow-up tests can be even more dramatically awesome. In the case of this client, who prefers to remain unnamed, the next round after an inconclusive result produced a triple-digit conversion rate lift.
Find Gold in the Losers
Like the ugly duckling that grows into a swan that lays golden eggs, you can analyze losers to gain insights. In other words, “There’s gold in them thar hills!“