The Definitive FAQ List for Conversion Rate Optimization

Month after month, we at WiderFunnel continue our free webinar series to show the best strategies for digital marketing success. We always have lively Q&A sessions at the end, where we answer your Conversion Optimization queries.

This post will become the start of an ongoing compilation of the common questions we’ve received. Chances are, you’ve had one or more of these questions too.

This is also a chance for you to add your own question at the bottom of this page to added to the list.

So, here goes the first 24 28 questions…

The (start of) the Ultimate Conversion Rate Optimization Frequently Asked Questions List

Q: If I’m running a paid search or an internet advertising campaign, should I take audiences to a one-page landing page, a mini microsite, or a page on my website? How does each method differ for lead generation?

This is a common question and it depends on the complexity of the purchase decision and whether they need the context of the rest of the site.

In most cases, we find one-page landing pages work best if they’re developed using proven conversion principles, but they also can be risky if the eyeflow and message isn’t well-crafted. For lead gen, typically less content is needed, but that’s not always the case. In one recent example, we tested a consumer lead gen landing page where the offer was a financial report for eco-conscious investing. We found that a long-copy landing page written by one of our hot copywriters gave a 91% conversion rate lift over a short page from a series of tests. This is a hypothesis we’ve often tested: “How much content is needed for prospects to make the decision to act?” You can test that by starting with a long copy landing page, then cutting it in half to test a mid-volume copy page, then go for a minimalist approach with the bare minimum content. You should test that!
Back to Top

Q: What are the key design elements in landing pages that contribute to increased conversions?

Here’s my answer:
a. First, the one thing that’s more important than tips and tricks for improving conversion rates is having a structured approach to continuous improvement. A recent research study by eConsultancy showed that “Companies with a structured approach to conversion are twice as likely to have seen a large increase in sales”
b. If I had to choose, I would say the three most important design elements are eyeflow, value proposition intro placement and call-to-action design.
c. If you think in terms of the LIFT Model that we use to analyze landing pages, the design of the page must maximize Clarity, Relevance and Urgency of the Value Proposition communication, while minimizing the Distraction and Anxiety.
d. The eyeflow is really affected by the combination of all design elements, though, so I’ve kind of cheated with that answer.
i. There are many things that influence the eyeflow, such as the wireframe or content layout, colours and imagery.
ii. A skilled conversion strategist will plan the visual stimulus to guide the visitor’s eye through the important messaging points and ultimately to the call to action.
e. The design of the call-to-action should maximize Clarity and Relevance.
Back to Top

Q: All the correct answers to the test examples you show in your webinars are against common sense. Could you get them just by going inverse psychology based?

(Asked by Teresa Montez)

WiderFunnel: Ha! Good question. What you’re asking is whether “best practices” are always wrong. No, the examples I showed in this webinar were just a few showing how common sense isn’t always right. They weren’t meant to indicate that common sense is always wrong. Plus if you would put these examples in a different context, you might just get completely different results. So: you should test that!

Psychology shows us how things should work. But, how things really work in specific situations is often entirely different. We recommend using psychological research as a basis for developing great test hypotheses. Then, test within your situation to find your specific Playbook of what really works.
Back to Top

Q: If our website receives large amount of repeat user who is used to a certain layout, when testing a drastic redesign, should we leave allow extra time for the user to become familiar with the new design? and how long should this “learning phase” be? Thanks Chris!

(Asked by Elaine Xu)

WiderFunnel: Regular web users (or “repeat visitors” in web analytics language) have become used to your current website, no matter how bad it is. It’s amazing how well people can adapt to using even the most terrible website!

The length of time in that “learning phase” will depend on your customers’ visit frequency. You can either run the test longer to allow customers time to learn the new design, or run the test separately on your new visitors.

Back to Top

Q: Is there a way to conclusively tell why a test lost or was inconclusive? To know what the hidden gold is?

(Asked by Jeff McKay)

Developing powerful hypotheses is dependent on having workable theories. Seeking to determine the “Why” behind the results is some of the most interesting part of the work.

The only way to tell conclusively is to infer a potential reason, then test again with new ways to validate that inference. Eventually, you can form conversion optimization theories and then test based on those theories. While you can never really know definitively the “why” behind the “what”, when you have theories and frameworks that work to predict results, they become just as useful.

As an example, I was reviewing a recent test for one of our clients and it didn’t make sense based on our LIFT Model. One of the variations was showing under-performance against another variations but I believed strongly that it should have over-performed. I struggled with this for some time to align this performance with our existing theories and eventually discovered the conversion rate listed was a typo! The real result aligned perfectly with our existing framework, which allowed me to sleep at night again!

By the way, here’s the reason why inconclusive and losing tests are actually a good thing.

Back to Top

Q: I might have missed it, but what are examples of heuristics?

(Asked by Scott MacMillan)

WiderFunnel: A heuristic is a kind of “rule of thumb” developed from experience, Scott. Our heuristic models, like the LIFT Model, use the knowledge gained from running many tests on different websites across many industries and types of companies. As I mentioned above, these models are developed based on many tests that lead to inferences, that lead to theories, that lead to models, that lead to predictive A/B test results.

Back to Top

Q: I’ve heard conversion consultants advise that you should start by testing as far down the conversion funnel as you can. Do you agree?

WiderFunnel: No! I’ve seen this advice bandied about and the rationale is askew. From a purely mathematical perspective, which portion of the funnel you start at makes no difference.

If you track your conversion goals properly, a 32.5% conversion rate lift at the top results in the same revenue lift as a 32.5% lift at the bottom end.

Here’s the math, in case you’re wondering:
x (1.325) * y = y (1.325) * x

So, there is no good mathematical argument to be made for starting at the top or the bottom.

The decision must be based on prioritizing the opportunities to improve conversion rates and revenue. And that depends on where the biggest problem areas are for the traffic segments on your website with the highest importance.

Here’s more on this topic: Should I start testing at the top or bottom of the website funnel?

Q: What is the name of the books Chris recommended in his recent webinar?

(Asked by Ray Ritchey)

WiderFunnel: There are many books we recommend to gain awesome conversion optimization knowledge. Here are a few to get you started:

Q: What are some of the software tools you mentioned that could help us do A/B testing?

(Asked by Greg Mischio)

WiderFunnel: You have many options to run A/B and multivariate tests on your web marketing today. Website testing tools generally fall under two categories: testing tools for existing websites or stand-alone landing page management apps.

Tools such as Visual Website Optimizer, Optimizely, Adobe Test & Target, SiteSpect, Hiconversion, and Maxymiser will enable testing on existing pages. They each have pros and cons, often with unique approaches to modifying your website content. Some simplify installation with easy JavaScript cut & paste, some need more in-depth installation for each test and others take over your websites domain name service altogether. The best option for your business depends on your company’s technical environment and conversion optimization maturity.

If you want to create new landing pages quickly that aren’t on your existing website, try a standalone landings page creation tool like Unbounce or Liveball.

Full disclosure: Chris Goward is an advisor to Unbounce and friend of many other testing tool companies.

Back to Top

Q: What testing tool would you recommend? We’ve were using Google Website Optimizer and now use the tool built into Google Analytics, but have had some really strange results. I’m looking for something more reliable.

(Asked by Ashley Ringrose)

WiderFunnel: Google Analytics is generally awesome. They’re continuously developing new killer features and it’s much more user-friendly than many of its competitors. Unfortunately, the Content Experiments tool within Google Analytics does not report results reliably. The multi-armed bandit algorithm is uses give misleading results. I don’t recommend it for A/B testing. See above for our recommended conversion optimization tools.

Back to Top

Q: What on-site survey tools you particularly recommend?

(Asked by Michał Kostrzewa)

Here are some of the leading on-site survey options to look into. They each have different sets of features and pros & cons. I’d suggest researching them in this order:

  • iPerceptions (which also bought 4Q):
  • Qualaroo:
  • Kampyle:
  • Foresee:

Back to Top

Q: Do you use mouse tracking software?

(Asked by Gijs Wierda)

WiderFunnel: Yes! We use CrazyEgg click-tracking software, and ClickTale or ScreenCam for detailed mouse-tracking and user interaction recordings. These tools allow our strategists to identify usability barriers on conversion funnel pages. We find these tools to be especially useful for transactional pages. They won’t show you ideas for the persuasional end of the conversion funnel, like on your landing pages, though. Here’s what we mean by the persuasional and transactional ends of the conversion funnel.

Special Offer! When CrazyEgg heard our plans to put together this article, they offered a special 60-day FREE CrazyEgg account for WiderFunnel readers You should try it! (Thanks Jordyn!)

Back to Top

Q: Are you confident that A/B testing tools don’t pose any risk to SEO?

(Asked by Star Bradshaw)

WiderFunnel: We’ve confirmed with Google that conducting split tests is not harmful to SEO. In fact, they encourage it. There are a few things to be aware of, however: if you’re using multiple URLs as part of an a/b/n test, use a canonical link tag that points back to your control URL. When done with the test, 301 your other URLs to the default version. Use a robots meta tag to noindex the URLs that are part of the test (except for your control variation). Here’s an article I posted about how to do conversion optimization with SEO.

Back to Top

Q: How many visits do you need to get to statistically relevant data of any individual test?

(Asked by Josh Becerra)

WiderFunnel: The number of visits is just one of the variables that determines statistical significance. The conversion rate of the Control and conversion rate delta between the variations are also part of the calculation. Statistical significance is achieved when there are enough conversions, conversion rate is high enough and the conversion rate delta is great enough. Here’s a handy Excel test duration calculator. Fortunately, today’s testing tools calculation statistical significance automatically, which simplifies the conversion champion’s decision-making (and saves hours of manual calculation!)

When planning tests, it’s helpful to estimate the test duration, but it isn’t an exact science. As a rule-of-thumb, we generally estimate 200-1000 conversions per variation to achieve statistical significance. Yes, that’s a wide range. Generally, more dramatic variations may need fewer conversions to reach significance.

Often, people that are new to testing become frustrated with tests that never seem to finish. If you’ve run a test with more than 30,000-50,000 visitors and one variation is still not statistically significant over another, then your test may not ever yield a clear winner and you should revise your test plan or reduce the number of variations being tested.

Back to Top

Q: Did you use EEG, fMRI or so while testing?

(Asked by Omer Gozu)

WiderFunnel: Brain-tracking technologies that you refer to can be used in what is called qualitative testing, which is where a group of individuals are asked to perform tasks on a website to gather data and perhaps come up with ways of improving a website. Qualitative testing is a step prior to the testing that we perform. It is one of the methods that can help a conversion strategist develop hypotheses. These methods can be used to infer the motivation behind people’s responses to stimuli and can lead to psychological theories and neuromarketing hypotheses. But, without quantitatively testing those theories and hypotheses in your environment, you can’t know how they’ll apply. It’s impractical to use brain scanning tools to monitor every change you make to a website, when an A/B test can be done quickly to validate the right decision.

The kind of testing WiderFunnel performs is done with quantitative (high volume) samples of real website visitors, to confirm these hypotheses. We use the people already visiting a website, from their home, their office, or smartphone. Our results are based on whether or not they make a purchase, download a brochure, or fill out a contact us form. In other words, where qualitative studies using tools like fMRI can give test ideas, they should be proven with qualitative samples of reach visitors and customers. There are many reasons that qualitative user testing alone is not enough for conversion optimization.

Back to Top

Q: What tools do you recommend for user testing?

Here are some great options for user testing:

  • Silverback for Mac
  • Morae for Windows

Back to Top

Q: Beyond the remote AB testings, do you meet your users (before or after) in order to understand the interpretation of your results ? Or is it mostly based on a guess on your side?

(Asked by Mathieu Bernimont)

WiderFunnel: All of our testing is done with a random sampling of users visiting the website, who we do not meet, but that we track through a statistical testing tool. We gather data on how tens of thousands of users interact with the website, depending on which variations in a test they see. The high volume of data that we receive makes for scientific conclusions, and requires no guesswork. The problem with asking someone what their interpretation of a piece of content is on a website is that many of the decisions that affect whether someone buys or does not buy are subconscious. A particular user may not be able to accurately describe why they did or did not buy as it happened in their brain without them consciously thinking about why.

Back to Top

Q: How exactly does WiderFunnel test using neuroscience?

(Asked by Diana Lucaci)

WiderFunnel: Brain-tracking technologies that you refer to can be used in what is called usability testing, which is where a group of individuals are asked to perform tasks on a website to gather data and perhaps come up with ways of improving a website. Usability testing is a step prior to the testing that we perform. It is where hypotheses are formed. The kind of testing we perform is done with real people, to confirm hypotheses. We use the people already visiting a website, from their home, their office, or smartphone. Our results are based on whether or not they make a purchase, download a brochure, or fill out a contact us form.

Back to Top

Q: When you run tests and gather data from customers, how do you align their needs with your goals? Are they in general aligned?

WiderFunnel: That’s one of the great outcomes of testing. You can learn how to beat align your goals with those of your prospects. If your message and goals aren’t aligned, they won’t convert, so you’ll learn very quickly if a different approach works better.

For example, one of our clients uses a freemium SaaS business model. They promote their free, limited version of their product on their landing pages with the goal of maximizing new user signups. Then, they have upgrade touchpoints and campaigns to convert the free users to one of the paid accounts. In our testing, we discovered that promoting the paid accounts could boost immediate paid user signup rather than having to go through the longer upswell process. And, it didn’t even sacrifice overall signups.

This showed that they should revise their goal to encompass paid signups to align with the segment of prospects who are ready to sign up for a paid account immediately. That testing resulted in a 58% revenue lift from one of their most important pages.

We’ve also posted many other conversion optimization case studies you should check out.

Back to Top

Q: When test results are insignificant after lots impressions, how do you know when to ‘call it a tie’ and stop that test and move on?

(Asked by Zach)

WiderFunnel: That’s a question that requires a large portion of “it depends.” It depends on whether:

  • You have other tests ready to run with the same traffic sources
  • The test results are showing high volatility or have stabilized
  • The test insights will be important for the organization

There’s and opportunity cost to every test. You could always be testing something else and need to constantly be asking whether this is the best test to be running now vs. the cost and potential benefit of the next test in your conversion strategy.

Back to Top

Q: how much traffic to a site do I need to properly do a/b testing?

WiderFunnel: It’s possible to test with as little as 1,000 monthly visitors, though you should be prepared for tests to take a very long time to complete with statistical significance. Above that, the more the merrier! Regardless of your traffic volume, there’s no downside to running a test. It can stay running for as long as you need. Even the longest-running test is better than none. When you have higher traffic volumes, your tests will complete more quickly and you can run more tests. And, with a higher traffic site, you should have more revenue to justify the added investment in creating more frequent tests. In general, you’ll be able to complete tests in a reasonable timeframe if you have more than 30,000 monthly visitors to a page.

Back to Top

Q: Can you guys evaluate an iPhone app?

(Asked by Victor)

WiderFunnel: Yes! The conversion optimization system WiderFunnel uses applies to all types of visitor environments. We optimize mobile apps, video game interfaces, and of course, websites, landing pages and checkout funnels.

Back to Top

Q: our value proposition is presented when visitors attempt to access “locked” features of our website. are there any insights can you share about this type of funnel?

WiderFunnel: websites with gated content, often subscription sites, make for very interesting tests. You have a small, focused area to communicate with maximum effect, which often makes for high elasticity tests. In other words, it’s a fun area to make a big impact! We’ve done lots of testing and have found thing beyond the tangible feature lists to be particularly important.

Back to Top

Q: Do you find strategies are differ cross-culturally? Do conversion rates vary drastically across different countries / languages when using these strategies?

WiderFunnel: we’ve run tests for many clients outside of the USA, such as in Israel, Sweden, Australia, UK, Canada, Japan, Korea, spain, Italy and for the Olympics store, which is itself a global ecommerce experience in one site! There are certainly cultural considerations and interesting differences in tactics. Some countries don’t have widespread credit card use, for example, and retailers there are accustomed to using alternative payment methods. Website design preferences in many Asian countries would seem very busy and overly colourful to a Western European visitor. WiderFunnel currently specializes in English-speaking and Western-European conversion optimization and work with partner CRO companies around the world to serve our global and international clients.

Back to Top

Q: how much traffic for multivariate testing?

(Asked by Igor)

WiderFunnel: For multivariate testing, we typically recommend a minimum of 100,000 monthly visitors to a test page to achieve statistical significance in a reasonable amount of time. But, more is better. In some cases, we won’t run MVT tests on a website with less than 250,000 per month. Most companies shouldn’t consider MVT until they’ve discovered a lot of the more accessible insights available with A/B/n testing.

Back to Top

Q: I work with someone who believes content is important. how do you successfully argue that writing an essay to explain a paragraph is actually dangerous?

WiderFunnel: Content is only valuable when it’s interesting. Content marketers and SEOs often fall into the same trap of measuring their success by the volume of content produced rather than the value of the content.

There’s a danger in over-emphasizing content at the expense of the real business goal. Content needs to generate revenue. If content is king, revenue is queen!

But, the best answer you can give to your colleague is “You should test that!” From all the tests we’ve run, it’s clear that communicating your message with fewer words is better for conversion rates. Your business is unique and more content may be needed, or a different kind of content. You should avoid endless debates by testing what works for you.

Back to Top

Q: How much value does the page fold have in today’s market – with different screen resolution and number of different devices accessing the content provide… how do you determine what the fold is?

(Asked by Craig Wishart)

WiderFunnel: The fold used to be easy to determine. You could just look at your primary visitor screen size – usually the majority of your visitors – then subtract 150 vertical pixels for toolbars.

Now, with the screen size continuum at play, life is more complex. You need to consider breakpoints and devices of infinite varieties.

But the fold is still very important. No matter how the scrolling mouse has improved the comfort level with long page scrolling, first impressions are critical, which we’ve shown time and time again with A/B tests.

Back to Top

Q: We provide core technology to help companies with an e-Commerce solution to increase their conversion rates via the use of video agents. Do you feel that face-to-face communication is still important in even the more mature e-Commerce markets?

(Asked by Jaz Sangha)

WiderFunnel: In cases where customers have many questions regarding a product, a video agent can be useful. Such a tool could also be used to conduct market research. You may be able to collect interesting bits of info about what your customers are interested in knowing about your products. In testing, we’ve found that the most influential pain points that an e-commerce shopper has are related to shipping costs, being ready to purchase the product (having enough desire), and wanting to compare prices at other websites. Video agents? You should test that!

Back to Top

What Conversion Optimization Questions Do You Have?

Add your questions below!

Leave A Comment