I love losing test results and you should too

| November 25, 2012

Recently, I gave a keynote presentation at the European Conversion Summit in Germany, and I revealed the best-kept secret of conversion optimization: Not all tests win.

Sadly, it’s true.

At WiderFunnel, we often share case studies of winning test results publicly. Most often, they show spectacular winning results and marketing insights.

awesome test results

While many people are understandably reluctant to acknowledge that they don’t always produce winners, I’m happy to. I’d like to tell you why.

Conversion Optimization Tests Don’t Always Win

I admit it. There are times that I’ve allowed conversion optimization test results to ruin my day.

You see, at WiderFunnel, we monitor every test on a daily basis so that we can make tactical changes to maximize ROI.

When there are tests on the dashboard that don’t show strong winning results, it’s hard not to let them get me down. I want to win big every time!

But We Do Win Most of the Time

Fortunately, the awesome team at WiderFunnel has an enviable success rate for tests.

I’ve had many conversations with internal testing teams and other agencies. Most companies report that around a third of their conversion optimization tests, out of every three, produce winners. An agency that runs tests regularly may get up to 50-60% wins, which is very good – about double the internal teams rate. With WiderFunnel’s system, we’re getting a much higher success rate. In our last quarter, 89% of tests planned by WiderFunnel produced a new champion page.


I should be thrilled. But… there’s still that 11 percent that didn’t deliver a new winner.

Take a look at these two test results as examples. Staring at these every morning, day after day… well, it gets old, fast.
What to do with inconclusive test results

Clearly a test with the big winner is more fun to see. But, an inconclusive test doesn’t annoy me quite as much as it used to.

Why I Love Inconclusive and Losing Test Results

Over the years, I’ve learned and re-learned that those losing and inconclusive tests hold valuable insights.

In fact, they can be so valuable that they should be called winners too.

We’ve seen over and over again that a commitment to continuous testing gets the best results. We often see the best results on the 2nd, 3rd and 4th tests on a page or template.

But, results don’t just come from repetition. Great results come from making insightful learning from each and every test. That’s important to understand: great results come from powerful insights.

Turn Inconclusive Tests into Winners

Where that inconclusive test I mentioned earlier would have ruined my day in years past, now our team looks at it as an opportunity.

We analyze the results of inconclusive tests to understand the differences and produce hypotheses that will get better results.

Analyze inconclusive tests to gain insights

In the case of this client with an inconclusive test, WiderFunnel’s next test round produced a statistically significant 16.1% winning result.

How to turn losing test into a winner

Sometimes the follow-up tests can be even more dramatically awesome. In the case of this client, who prefers to remain unnamed, the next round after an inconclusive result produced a triple-digit conversion rate lift.

Analyze inconclusive tests for learning

Follow up from inconclusive test gives huge win

Find Gold in the Losers

Like the ugly duckling that grows into a swan that lays golden eggs, you can analyze losers to gain insights. In other words, “There’s gold in them thar hills!

If you find this interesting please share with your friends or colleagues:

  • Muhammad Ali

    Thanks for Sharing Case Studies Chris.

    • http://twitter.com/chrisgoward @chrisgoward

      You're welcome, Muhammad! I hope you've found them useful.

  • shree

    hi chris.love your site.These conversion information has been really helpful for some of my businesses.In a marketing standpoint , how would you promote a conversion optimisation service in a country specific environment that has only been used to ppc experts?

    • http://twitter.com/chrisgoward @chrisgoward

      You may be in a similar situation to where we were in 2007: selling the concept of conversion optimization when very few were doing it. It takes time and you may need to start with smaller test projects to prove the concept.

      The advantage to being an early mover is that you can learn more and be a leader in the future industry.

      If you only do CRO, you could try partnering with PPC experts and other agencies, like we do.