Request a Free Landing Page Evaluation

The Marketing Optimization Blog

Turning More Visitors Into Customers

Blog Home

Are you sure your test results are valid?

By: Chris Goward
Date: September 3rd, 2007

There is one big problem with Google offering free products, like Google Website Optimizer and Google Analytics.

No, it’s not the product quality issue that you normally find when a company competes on price. Both Website Optimizer and Google Analytics are robust, enterprise-grade products. Although they don’t have all the best features available in some competitors, if you’re using the tools properly to their full potential, you’re doing better than most that pay for their website optimization and analytics tools.

The biggest benefit, that the tools are accessible to virtually anyone with a website, is also the biggest problem. Anyone with a passing interest in analytics can install the tools and quickly come to some interesting, and often invalid, conclusions.

There are surprisingly simple ways to draw conclusions from a/b tests or multivariate tests that will lead you to make disastrous decisions for your web site.

For example, the most basic rule of scientific methodology is that you NEVER touch the “control” during an experiment. This, unfortunately, comes as a surprise to many people that want to start testing online.

There are numerous other validity risks that must be accounted for, such as measurement tool assumptions, seasonality fluctuations, traffic segmentation factors, historical factors, among others.

Unlike a road driving test, which will give you immediate and obvious feedback when you’ve made an error (ie. your car is wrapped around a pole), an online testing error is careful to conceal itself in the numbers.

The question is: Are you sure your test results are valid?

Ask Us How to Improve Your Conversion Rate

WiderFunnel is the original full service Conversion Optimization agency. When you outsorce you conversion improvement to us you get guaranteed results with an all-inclusive price.

Posted in: best practices

This website uses IntenseDebate comments, but they are not currently loaded because either your browser doesn't support JavaScript, or they didn't load fast enough.

One Response to “Are you sure your test results are valid?”

  1. Melina Says:
    December 20th, 2007 at 11:59 am

    very interesting. i’m adding in RSS Reader


Leave a Comment



.