I love losing test results and you should too

3 min. read | Last updated: November 20th, 2015

Recently, I gave a keynote presentation at the European Conversion Summit in Germany, and I revealed the best-kept secret of conversion optimization: Not all tests win.

Sadly, it’s true.

At WiderFunnel, we often share case studies of winning test results publicly. Most often, they show spectacular winning results and marketing insights.

awesome test results

While many people are understandably reluctant to acknowledge that they don’t always produce winners, I’m happy to. I’d like to tell you why.

Conversion Optimization Tests Don’t Always Win

I admit it. There are times that I’ve allowed conversion optimization test results to ruin my day.

You see, at WiderFunnel, we monitor every test on a daily basis so that we can make tactical changes to maximize ROI.

When there are tests on the dashboard that don’t show strong winning results, it’s hard not to let them get me down. I want to win big every time!

But We Do Win Most of the Time

Fortunately, the awesome team at WiderFunnel has an enviable success rate for tests.

I’ve had many conversations with internal testing teams and other agencies. Most companies report that around a third of their conversion optimization tests, out of every three, produce winners. An agency that runs tests regularly may get up to 50-60% wins, which is very good – about double the internal teams rate. With WiderFunnel’s system, we’re getting a much higher success rate. In our last quarter, 89% of tests planned by WiderFunnel produced a new champion page.

Awe-some!

I should be thrilled. But… there’s still that 11 percent that didn’t deliver a new winner.

Take a look at these two test results as examples. Staring at these every morning, day after day… well, it gets old, fast.
What to do with inconclusive test results

Clearly a test with the big winner is more fun to see. But, an inconclusive test doesn’t annoy me quite as much as it used to.

Why I Love Inconclusive and Losing Test Results

Over the years, I’ve learned and re-learned that those losing and inconclusive tests hold valuable insights.

In fact, they can be so valuable that they should be called winners too.

We’ve seen over and over again that a commitment to continuous testing gets the best results. We often see the best results on the 2nd, 3rd and 4th tests on a page or template.

But, results don’t just come from repetition. Great results come from making insightful learning from each and every test. That’s important to understand: great results come from powerful insights.

Turn Inconclusive Tests into Winners

Where that inconclusive test I mentioned earlier would have ruined my day in years past, now our team looks at it as an opportunity.

We analyze the results of inconclusive tests to understand the differences and produce hypotheses that will get better results.

Analyze inconclusive tests to gain insights

In the case of this client with an inconclusive test, WiderFunnel’s next test round produced a statistically significant 16.1% winning result.

How to turn losing test into a winner

Sometimes the follow-up tests can be even more dramatically awesome. In the case of this client, who prefers to remain unnamed, the next round after an inconclusive result produced a triple-digit conversion rate lift.

Analyze inconclusive tests for learning

Follow up from inconclusive test gives huge win

Find Gold in the Losers

Like the ugly duckling that grows into a swan that lays golden eggs, you can analyze losers to gain insights. In other words, “There’s gold in them thar hills!

Author

Chris Goward

Founder & CEO

Discover how your experimentation program stacks up!

Benchmark your experimentation maturity with our new 7-minute maturity assessment and get proven strategies to develop an insight-driving growth machine.

Get started