The duration of a test is one of the most frequent questions we hear from our clients. It can be frustrating to see the time remaining estimate fluctuate, or worse, steadily increase as your test progresses.
Unfortunately, there is no simple answer.
The number of test combinations and traffic level are the variables that you can calculate. And Google has provided a calculator that does a ballpark job for you.
However, there are variables to reaching statistical significance that can’t be predicted, which are: 1. The conversion rate of the control 2. The spread between the conversion rate of the control and the alternative/test
If the conversion rate is high and the spread between the control and the variations are large, the test will complete relatively quickly. If, however, the test alternatives are very similar in conversion rates to the control, it will take much longer to reach statistical significance.
That is why our advice to clients is to let us design dramatic tests rather than piddling around with a word here and there. There’s much more upside to be had in being bold with the variations.
It’s also tempting to stop an experiment before it’s complete. And that’s fine to do if it’s not completing fast enough and you want to modify it and try again. But be careful in taking learning from an incomplete test. It’s easy to infer too much into all that green on the bar when it is really not ready to tell you anything.
This website uses IntenseDebate comments, but they are not currently loaded because either your browser doesn't support JavaScript, or they didn't load fast enough.
Name*
Mail* (Will not be published)
Website
Blog-To-Email Approx. 1-2 per week.
Our Conversion Optimization Experts are ready to work with you to measurably improve your business. Call Us Now: 604-569-0897
Download Conversion Rate Optimization whitepapers, ROI calculators, and worksheets. Call Us Now: 604-569-0897