Are you sure your test results are valid?

1  | Last updated: March 12th, 2015

There is one big problem with Google offering free products, like Google Website Optimizer and Google Analytics.

No, it’s not the product quality issue that you normally find when a company competes on price. Both Website Optimizer and Google Analytics are robust, enterprise-grade products. Although they don’t have all the best features available in some competitors, if you’re using the tools properly to their full potential, you’re doing better than most that pay for their website optimization and analytics tools.

The biggest benefit, that the tools are accessible to virtually anyone with a website, is also the biggest problem. Anyone with a passing interest in analytics can install the tools and quickly come to some interesting, and often invalid, conclusions.

There are surprisingly simple ways to draw conclusions from a/b tests or multivariate tests that will lead you to make disastrous decisions for your web site.

For example, the most basic rule of scientific methodology is that you NEVER touch the “control” during an experiment. This, unfortunately, comes as a surprise to many people that want to start testing online.

There are numerous other validity risks that must be accounted for, such as measurement tool assumptions, seasonality fluctuations, traffic segmentation factors, historical factors, among others.

Unlike a road driving test, which will give you immediate and obvious feedback when you’ve made an error (ie. your car is wrapped around a pole), an online testing error is careful to conceal itself in the numbers.

The question is: Are you sure your test results are valid?


Chris Goward

Founder & CEO

Discover how your experimentation program stacks up!

Benchmark your experimentation maturity with our new 7-minute maturity assessment and get proven strategies to develop an insight-driving growth machine.

Get started