I Love Losing Test Results and You Should Too

Chris Goward Conversion Optimization not all tests win
Recently, I gave a keynote presentation at the European Conversion Summit in Germany, and I revealed the best-kept secret of conversion optimization: Not All Tests Win.   Tweet this:

The best-kept secret of conversion optimization: Not All Tests Win

Sadly, it’s true.

At WiderFunnel, we often share case studies of winning test results publicly. Most often, they show spectacular winning results and marketing insights.

awesome test results

While many people are understandably reluctant to acknowledge that they don’t always produce winners, I’m happy to. I’d like to tell you why.

Conversion Optimization Tests Don’t Always Win

I admit it. There are times that I’ve allowed conversion optimization test results to ruin my day.

You see, at WiderFunnel, we monitor every test on a daily basis so that we can make tactical changes to maximize ROI.

When there are tests on the dashboard that don’t show strong winning results, it’s hard not to let them get me down. I want to win big every time!

But We Do Win Most of the Time

Fortunately, the awesome team at WiderFunnel has an enviable success rate for tests.

I’ve had many conversations with internal testing teams and other agencies. Most companies report that around a third of their conversion optimization tests, out of every three, produce winners.   Tweet this:

For most companies, only 1/3 of CRO tests produce winners
An agency that runs tests regularly may get up to 50-60% wins, which is very good – about double the internal teams rate. With WiderFunnel’s system, we’re getting a much higher success rate. In our last quarter, 89% of tests planned by WiderFunnel produced a new champion page.   Tweet this:

89% of CRO tests planned by WiderFunnel produced a winner. Awesome!


I should be thrilled. But… there’s still that 11 percent that didn’t deliver a new winner.

Take a look at these two test results as examples. Staring at these every morning, day after day… well, it gets old, fast.
What to do with inconclusive test results

Clearly a test with the big winner is more fun to see. But, an inconclusive test doesn’t annoy me quite as much as it used to.   Tweet this:

Why inconclusive tests don't annoy me as much as they used to

Why I Love Inconclusive and Losing Test Results   Tweet this:

Why I Love Inconclusive and Losing Conversion Optimization Test Results

Over the years, I’ve learned and re-learned that those losing and inconclusive tests hold valuable insights.   Tweet this:

Losing and inconclusive conversion optimization tests hold valuable insights

In fact, they can be so valuable that they should be called winners too.   Tweet this:

Why losing and inconclusive conversion optimization tests should be called winners

We’ve seen over and over again that a commitment to continuous testing gets the best results. We often see the best results on the 2nd, 3rd and 4th tests on a page or template.

But, results don’t just come from repetition. Great results come from making insightful learning from each and every test. That’s important to understand: great results come from powerful insights.   Tweet this:

Great conversion optimization results come from powerful insights

Turn Inconclusive Tests into Winners

Where that inconclusive test I mentioned earlier would have ruined my day in years past, now our team looks at it as an opportunity.

We analyze the results of inconclusive tests to understand the differences and produce hypotheses that will get better results.

Analyze inconclusive tests to gain insights

In the case of this client with an inconclusive test, WiderFunnel’s next test round produced a statistically significant 16.1% winning result.

How to turn losing test into a winner

Sometimes the follow-up tests can be even more dramatically awesome. In the case of this client, who prefers to remain unnamed, the next round after an inconclusive result produced a triple-digit conversion rate lift.   Tweet this:

The follow up from inconclusive test can give triple-digit win

Analyze inconclusive tests for learning

Follow up from inconclusive test gives huge win

Find Gold in the Losers

Like the ugly duckling that grows into a swan that lays golden eggs, you can analyze losers to gain insights. In other words, “There’s gold in them thar hills!   Tweet this:

There's gold in them thar hills!


  1. Muhammad Ali December 18, 2012 at 11:55 pm - Reply

    Thanks for Sharing Case Studies Chris.

    • @chrisgoward December 19, 2012 at 12:15 pm - Reply

      You're welcome, Muhammad! I hope you've found them useful.

  2. shree December 30, 2012 at 3:10 pm - Reply

    hi chris.love your site.These conversion information has been really helpful for some of my businesses.In a marketing standpoint , how would you promote a conversion optimisation service in a country specific environment that has only been used to ppc experts?

    • @chrisgoward December 31, 2012 at 11:29 am - Reply

      You may be in a similar situation to where we were in 2007: selling the concept of conversion optimization when very few were doing it. It takes time and you may need to start with smaller test projects to prove the concept.

      The advantage to being an early mover is that you can learn more and be a leader in the future industry.

      If you only do CRO, you could try partnering with PPC experts and other agencies, like we do.

Leave A Comment