The duration of a test is one of the most frequent questions we hear from our clients. It can be frustrating to see the time remaining estimate fluctuate, or worse, steadily increase as your test progresses.

Unfortunately, there is no simple answer.

The number of test combinations and traffic level are the variables that you can calculate. And Google has provided a calculator that does a ballpark job for you.

However, there are variables to reaching statistical significance that can’t be predicted, which are:
1. The conversion rate of the control
2. The spread between the conversion rate of the control and the alternative/test

If the conversion rate is high and the spread between the control and the variations are large, the test will complete relatively quickly. If, however, the test alternatives are very similar in conversion rates to the control, it will take much longer to reach statistical significance.

That is why our advice to clients is to let us design dramatic tests rather than piddling around with a word here and there. There’s much more upside to be had in being bold with the variations.

It’s also tempting to stop an experiment before it’s complete. And that’s fine to do if it’s not completing fast enough and you want to modify it and try again. But be careful in taking learning from an incomplete test. It’s easy to infer too much into all that green on the bar when it is really not ready to tell you anything.

Okay, this one is really weird.

My wife has recently become a fan of the kids clothes at NEXT, a UK clothing and household store.

But it’s got some very strange quirks that I can’t imagine the rationale for:

• The online shopping cart closes every day without warning. Seriously, they’ve got closing hours.. on the Internet…
• When the shop closes, your shopping cart automatically empties.
• If you leave your shopping cart for more than 1/2 hour, it empties.

It wouldn’t take a usability expert or months of focus group testing to tell how you can increase your conversion rate.

Maybe I won’t tell you, though. At least it keeps my wife from giving you all my money.

WiderFunnel Rating:

We got the official media release from Omniture this morning: Omniture Acquires Offermatica

We often get asked about the differences between an analytics tool and an optimization tool. If anyone still seriously questions the value of optimization, this acquisition demonstrates it. The answer: $65 Million.

As we always say, the analytics tool is for diagnosing the problem and the optimization tool is for finding the solutions. We at WiderFunnel have experience working with both Omniture and Offermatica – and welcome the acquisition as a great step forward in improving marketing effectiveness.

The tighter integration of these tools is great news for Omniture and Offermatica customers!

As the Google Analytics Blog announced this morning, Website Optimizer just got an update.

The A/B testing Wizard is the most significant advance. Now, it’s possible to easily test dramatic landing page variations without worrying about the restrictions on the types of code that can be within multivariate variables.

Perfect timing for a client A/B/n test we’ll be launching shortly. Thanks Google!

I’m a big fan of Google Website Optimizer and we’ve found that most of our clients, not surprisingly, are enjoying the free price (and when I say “most”, it doesn’t mean that some don’t like the free price, just that they want to do more advanced testing that’s more practical with higher priced alternatives).

One issue I have, though, is with the 30-minute, single session cookie restriction. This rule means that Google Website Optimizer will only count a conversion if it occurs within 30 minutes and within the same session of viewing the test page.

So, if you visited my test page and saw one of 48 variable combinations, for example, and then 30 minutes later returned to the site to complete the transaction, you have a 98% probability (48/49, including the control) of seeing a different page combination. The problem is that the last combination is the only one that gets credited for the conversion, which shouldn’t necessarily be so.

This seems to be artificially oriented toward immediate impulse actions and may skew results.

I’ve passed this feedback on to Tom Leung (Google Website Optimizer Product Manager), and hope that they’ll be able to incorporate a change in the next Optimizer version, which I’m eagerly awaiting, I should add!

Ideally, we should be able to easily set our own rules for the timespan and persistence of the cookie based on the clients’ particular business rules and product purchase cycle.

There is one big problem with Google offering free products, like Google Website Optimizer and Google Analytics.

No, it’s not the product quality issue that you normally find when a company competes on price. Both Website Optimizer and Google Analytics are robust, enterprise-grade products. Although they don’t have all the best features available in some competitors, if you’re using the tools properly to their full potential, you’re doing better than most that pay for their website optimization and analytics tools.

The biggest benefit, that the tools are accessible to virtually anyone with a website, is also the biggest problem. Anyone with a passing interest in analytics can install the tools and quickly come to some interesting, and often invalid, conclusions.

There are surprisingly simple ways to draw conclusions from a/b tests or multivariate tests that will lead you to make disastrous decisions for your web site.

For example, the most basic rule of scientific methodology is that you NEVER touch the “control” during an experiment. This, unfortunately, comes as a surprise to many people that want to start testing online.

There are numerous other validity risks that must be accounted for, such as measurement tool assumptions, seasonality fluctuations, traffic segmentation factors, historical factors, among others.

Unlike a road driving test, which will give you immediate and obvious feedback when you’ve made an error (ie. your car is wrapped around a pole), an online testing error is careful to conceal itself in the numbers.

The question is: Are you sure your test results are valid?

We are very excited to announce that Google has published a case study based on an early conversion optimization project we completed with our client, Tourism British Columbia.

It is now available on the Google Website Optimizer home page for a limited (but hopefully not short!) time. Its permanent location is in the case studies page.

Yay!

Thanks to Tom Leung and Peter Harbison for making it happen. Cheers guys!