5 test results that made us say ‘A-ha!’ in 2016

Reading Time: 10 minutes

‘A-ha!’ moment (n.): An insight that leads to more substantial revenue lift and profitable growth for your company (e.g. the moment all Optimizers live for).

At WiderFunnel, our mission is create profitable ‘A-ha!’ moments for our clients every day.

Last year, I created a five-part ‘A-ha!’ moments series: Five mini blog posts focused on five of our favorite insights from 2015. Well, turns out 2016 was also full of ‘A-ha!’ moments that were too good to keep to ourselves.

This post explores five of WiderFunnel’s favorite ‘A-ha!’s from the past year. I hope that they inspire you as you begin planning your 2017 experiments!

Want to have more 'A-ha!' moments of your own?

Get 17 A/B test ideas from WiderFunnel's Optimization Experts, plus where to test each idea and wireframes to guide you!

‘A-ha!’ #1: Using color psychology to increase conversions

If you follow WiderFunnel, you probably know that we are not big fans of conversion optimization ‘best practices’ like “all calls-to-action should be orange”.

Because, frankly, best practices may not be the best thing for your business. They must be proven in your business context, for your users.

That said, this first ‘A-ha!’ moment comes from a color isolation test. But, the ‘A-ha’ isn’t the result, it’s the why behind the hypothesis.

The strategy

One of our clients provides an online consumer information service — users type in a question and get an Expert answer. Once a user asks their question, they have entered a four-step funnel:

  • Step 1: Ask the question
  • Step 2: Add more information
  • Step 3: Pick an Expert
  • Step 4: Get an answer (aka the checkout page)

We have been testing on each step of this funnel, but this particular experiment was on the all-important checkout page, the final conversion.

What can the right color do?

For each WiderFunnel client, we create a customized growth program, however, each program is built with our proven Infinity Optimization Process™. The process cycles between two phases: Explore (information-gathering) and Validate (testing and proving).

Research on consumer behavior, psychological principles, and persuasion techniques is a huge part of the Explore phase. Our Strategists use this research, along with several other information touchpoints, when developing hypotheses.

This past year, one of WiderFunnel’s favorite bloggers and researchers, Nick Kolenda, published a giant piece on color psychology. Kolenda looked at 50 academic studies on color, and compiled his findings. According to him, certain colors can inspire certain actions.

Aha! #1 color spectrum
Can certain colors influence your users’ behavior?

In the case of this client, Optimization Strategist, Nick So, wanted to see if adding a subtle, subconscious visual cue to the checkout page would be more motivational for users. He was looking, specifically, at warm colors.

Persuasion principle
: Warm colors (with high saturation and low brightness) increase arousal because they trigger impulsivity, and tend to increase behavioral responses.

The test: Isolation I and isolation II

In the first isolation, Nick decided to put warm colors to the test.

Hypothesis: Increasing prominence of the checkout area by using a color linked to increasing action and responses will improve visual clarity of the page and increase conversions.

Aha! #1 Control
The client’s original checkout page.
Aha! 1 VarA
Our variation, which emphasized the payment section with a warm color background.

In the variation, Nick removed all other background colors and added a warm orange background to the payment section. And it worked! This variation saw a statistically significant 2.82% increase in conversions.

We wanted to validate this insight across audiences, so Nick created a second isolation for this client’s mobile users.

Aha! #1 mobile
From right to left: the Control, VarA, and the winning VarB.

He tested the Control against two variations: Variation B (the warm color isolation) was built on variation A, so Nick was able to track the isolation properly. In this experiment, the color change was responsible for a 2.7% lift in conversions, almost the exact same increase as in the desktop test.

A-ha!

Nick So WiderFunnel

It’s always amazing how such seemingly subtle psychological cues and persuasion elements can have a big potential impact on user behavior. We are fortunate to be able to have a client that has the traffic, trusts us, and understands testing enough to allow us to run an isolation on such an interesting concept.

– Nick So

‘A-ha!’ #2: Sometimes, all your users need is a clear next step

You may have heard the phrase “if content is king, revenue is queen”…

WiderFunnel Founder & CEO, Chris Goward, wrote, “Content is important for getting people to your site, from search algorithms to social share to links to your site, but content alone doesn’t make you revenue. Content without conversions is just free publishing.

Our second ‘A-ha!’ moment comes from testing we have been doing with one WiderFunnel client: A content site that provides information for the individual investor. This client offers a ton of free resources on its website to help users stay on top of their finances.

Of course, they also offer subscription services, such as their newsletter and professional advisor service, which provides premium stock-picking advice to users. Our goal is to help this client increase profitable conversions.

The strategy

When we began testing with this client, there were many different paths that users could take after landing on an investing article. And there was almost no indication that there were professional services available (which is how this client makes money!)

The WiderFunnel Strategy team did an initial LIFT analysis of the site-wide navigation, which revealed several problems, like:

  • There was not a clear, primary call-to-action in the nav (Clarity)
  • There was a general lack of urgency (Urgency)
  • The menu drop-down for “Stock Picks” had one, ambiguous dropdown (Anxiety)
  • If someone is ready to spend money, it is not clear how to do so (Clarity)
Aha! #2 Control
The original navigation didn’t have a clear call-to-action.

We wanted to test giving users a clear action to take in the site-wide navigation. This way, a user who wanted more would know which path to take.

We tested adding a “Latest Stock Picks” call-to-action in the nav (replacing the “Stock Picks” dropdown); the assumption was that users of this client’s site are looking for stock-picking advice, specifically.

Hypothesis: Creating a clear “Latest Stock Picks” CTA in the site-wide navigation will cause more users to enter a revenue-driving funnel from all parts of the site.

The variations

We tested two variations, each of which featured the “Latest Stock Picks” call-to-action. But, in each variation this CTA took the user to a different page. Our ultimate goal was to find out:

  1. If users were even aware that there are premium paid services offered, and
  2. Which funnel is best to help users make a decision and, ultimately, a purchase?

With variation A, we added the “Latest Stock Picks” CTA in the nav. This call-to-action sent users to the homepage and anchored them in the premium services section. (This is how the functionality of the original dropdown worked.)

This section provides a lot of detail about this client’s different offerings, along with a “Sign Up Today” call-to-action.

Aha! #2 VarA
The winning variation featured a very clear call-to-action, while maintaining the same functionality as the Control.

With variation B, we wanted to test limiting choice. Rather than showing users a bunch of product options, the “Latest Stock Picks” CTA sent them directly to the professional advisor sign up page (this client’s most popular product).

Aha! #2 VarB
In this variation, the CTA sent users to a product page.

A-ha!

Both variations beat the control, with variation A resulting in an 11.17% lift in transactions with 99% confidence and variation B resulting in a 7.9% increase in transactions with 97% confidence.

Interestingly, because variation B was built on variation A, we were able to see that it actually decreased transactions by 3.3%.

So, what does this mean? Here are a few takeaways we plan to explore further in 2017:

  • Users may have been unsure of how to sign up (or that they could sign up) due to lack of CTA prominence on the original site-wide navigation
    • It is also possible that Urgency was a motivator for this client’s users: Changing the “Stock Picks” drop down to a “Latest Stock Picks” CTA increased urgency and led to more conversions. This wasn’t a clear isolation but it’s good evidence to follow-up with!
  • Users prefer some degree of choice over being sent to one product (as seen with the decrease in transactions caused by variation B)

But the main moral of this ‘A-ha!’? Make sure your users know exactly where to find what you’re selling. ‘Cause content without conversions is just free publishing.

‘A-ha!’ #3: The power of proper Design of Experiments

Earlier this year, I published a case study on WiderFunnel client, weBoost. WeBoost is an e-commerce retailer and manufacturer of cellular signal boosters.

This case study explored several tests that we had run on multiple areas of the weBoost site, including a series of design tests we ran on their product category page. Our third ‘A-ha!’ moment takes up where the case study left off in this series…

A quick refresher

Originally, the weBoost product category pages featured a non-traditional design layout. A large image in the top left corner, very tall product modules, and right-hand filters made these pages unique among e-commerce catalog pages.

Aha! #3 Original
The original product category page layout.

We decided to test displaying products in landscape versus the long, portrait-style modules. According to a Baymard study of e-commerce sites, technical products are easier to compare in a horizontal layout because there is more space to include specs. This was variation A.

Aha! #3 Horizontal
Variation A featured a simple change: vertical modules to horizontal.

In variation B, we wanted to explore the idea that users didn’t need to see a product details page at all. Maybe the information on the category page was all users needed to make a confident purchase.

Variation B was built on variation A, with one isolated change: We changed the primary visual call-to-action from “View Details” to “Add To Cart”.

Aha! #3 Add To Cart
Note the primary CTA in this variation: “Add To Cart”

In a backward ‘A-ha!’ moment, variation A (based on the Baymard study) decreased transactions by -9.6%. Despite our intentions, the horizontal layout might have made it more difficult for users to compare products.

But! Variation B, with the add-to-cart focus, saw a 16.4% increase in transactions against the control page. It turns out that many users are actually comfortable adding products to their cart right from the category page.

Variation B moved more users further through the funnel and ultimately resulted in a large uptick in transactions, despite the negative impact of the horizontal layout.

After comparing variation A to variation B, WiderFunnel Optimization Strategist, Michael St Laurent, estimated that the “Add To Cart” call-to-action was actually worth a lift of 28.7% in transactions.

The follow-up (and subsequent ‘A-ha!’)

We knew that the horizontal layout led to a decrease in transactions and we knew that the horizontal layout plus the isolated CTA change led to a sizable increase in transactions.

So, we ran the obvious follow-up experiment: We tested a variation featuring the vertical module design with the add-to-cart focused call-to-action. We expected to see at least a 29% increase in transactions. We used variation B from the previous test as the Control, following proper Design of Experiments.

Aha! #3 Final
This variation reverted to the vertical modules from the original page, and featured the “Add To Cart” CTA.

As predicted, when we tested the “Add To Cart” call-to-action on the vertical modules, we saw a whopping 38.1% increase in transactions (more than double the 16.4% increase we observed with the horizontal layout, and 9 percentage points more than the estimate).

A-ha!

It never gets old to see isolations at work. The ‘A-ha!’ moment here is that no test ever has to be a ‘loser’. If you structure your tests using isolations, you will be able to track the potential impact of each change.

Michael St Laurent WiderFunnel

This entire time, we were assuming that users needed more information to make a technical product selection. We were focused on making the specs easier to compare, when there was an entire segment of the audience that was ready to put the product in their cart without more investigation. Sometimes you have to challenge your assumptions. In this case it paid off!

– Michael St Laurent, Optimization Strategist, WiderFunnel

‘A-ha!’ #4: De-emphasizing price reduces user anxiety

One of our clients is Vital Choice, a trusted source for fast home delivery of the world’s finest wild seafood and organic fare, harvested from healthy, well-managed wild fisheries and farms.

Our fourth ‘A-ha!’ moment from 2016 came out of the testing we did with Vital Choice on their product detail pages and revolves around de-emphasizing price, in favor of value proposition points.

While the results may not be surprising, the WiderFunnel Strategy team would not have prioritized this particular test if they hadn’t done extensive user research beforehand. Because we took the pulse of Vital Choice users, we were able to reduce anxiety and provide more motivation to purchase.

The strategy

Let’s say you wanted to order a few organic, grass-fed American Wagyu beef patties from the Vital Choice website. You would have eventually landed on a detail page that looked like this (the Control in this experiment):

Aha! #4 Control
Note the prominence of price on the original detail page.

As you can see, price is displayed prominently near the ‘Add To Cart’ call-to-action. But, during the Explore (information gathering) phase, WiderFunnel Optimization Strategist, Dennis Pavlina, identified several common themes of barriers to conversion in user survey responses, including:

  1. Price: Users love Vital Choice and the excellent quality of their products, but they often mention the premium they are paying. For many users, it is a ‘treat’ and a ‘luxury’ to buy from Vital Choice. Price-related themes, such as discount codes or coupons, also came up often in surveys.
  2. Shipping: Users often express concern about how frozen perishable items are shipped, particularly in warmer climates in the U.S.

If we could reduce user anxiety in these two areas, we believed Vital Choice would see a surge in conversions.

The test

Hypothesis: Adding relevant value proposition points that justify the price and quality of the product, and adding copy to reduce anxiety around shipping in close proximity of the order area on the product page, will increase conversions.

With our variation, we wanted to address the following barriers to conversion on the original page:

  • It was unclear what users would receive in their shipment i.e. how it would be shipped to them, how long it would take, etc. (Anxiety)
  • There were no prominently displayed value proposition points to justify the price of the product. (Value Proposition)
  • There was a lot of emphasis on the price of the product. (Anxiety)
Aha! #4 VarA
This variation addressed user anxieties by de-emphasizing price, and reassuring users of shipping guarantees.

A-ha!

This variation led to a 3.3% increase in conversions and a 2.7% increase in average order value, resulting in almost $250,000 in estimated additional annual revenue.

Conversions were up for almost every goal we tracked: Visits to checkout (step 2), visits to checkout (step 3), visits to checkout (step 4), total visits to cart, and average order value. But they were down to unique visits to cart.

Dennis Pavlina WiderFunnel

The most interesting part of analyzing results was noticing that, although unique visits to cart were slightly down, there was a large increase in total visits to cart. It’s a surprising pattern. We hypothesize that users may have been more confident and willing to purchase more items at once, when anxiety was reduced.

– Dennis Pavlina, Optimization Strategist, WiderFunnel

The fact that de-emphasizing price worked for Vital Choice users isn’t what made us say, ‘A-ha!’. But, the proven power of listening to, and addressing their users’ stated concerns, did. When in doubt, ask your users.

‘A-ha!’ #5: Quick view, long delay

‘A-ha!’ number 5 comes from testing we did with another one of our clients, a large retailer of sports goods, footwear, and apparel. We have been working with this company for more than a year to optimize their e­-commerce experiences, with the goal of increasing transactions.

Like on many e-commerce sites, users on this client’s site could view product details directly on the category page, using a Quick View functionality. When a user hovered over a product, they would see the product details in a Quick View window.

In our final ‘a-ha!’, we explore what (so often) happens when you test a common practice.

The strategy

Distraction is a very common barrier to conversion; often, there are elements on a client’s page that are diverting visitors away the from the ultimate goal.

For Michael St Laurent, the Quick View option on this client’s category page was a potential distraction.

Michael St Laurent WiderFunnel

The more visual cues and action options your visitor has to process, the less likely they are to make a conversion decision. At WiderFunnel, we have found that minimizing distractions such as unnecessary product options, links, and extraneous information will increase your conversion rate.

– Michael St Laurent

So, he decided to put his theory that the Quick View is an unnecessary distraction to the test.

The test

Hypothesis: Disabling the Quick View functionality will result in reduced distraction and ultimately, more conversions.

The Control in this test was the client’s original category page, featuring the Quick View functionality.

Aha! #5 Control
The original Quick View functionality.

In the Quick View, users could quickly move from product to product on the category page without going to a product page itself.

We tested this control against a variation that removed the Quick View functionality completely.

Aha! #5 No Quick View
In our variation, we eliminated the Quick View functionality entirely.

A-ha!

It turns out the Quick View functionality was, indeed, distracting. Disabling it resulted in more product exploration as well as more transactions; transactions increased by 4% (a big lift for a high-traffic company like this one!)

If your site has a functionality, like Quick View or a rotating banner, you should probably test it! While ‘flashy’ functionalities are…well…flashy, they are rarely what your users want, and may be preventing your users from actually purchasing.

At the end of every month, the WiderFunnel Strategy team shares their favorite ‘A-ha!’ moments from the past four weeks. Sometimes, the ‘A-ha!’ is an exciting result and big lift for a client, sometimes it’s a twist insight, sometimes it’s a ‘losing’ test that inspired a winning test.

As Chris Goward explains,

There’s no downside to communicating what you’ve learned from every test. If you view your optimization program as a strategic method for learning about your customers and prospects – for truly understanding their mindset – rather than a tactical tweaking program, you can take a broader perspective and find the gains in every test.

I hope that these ‘A-ha!’ moments inspire you to do the work, structure your tests properly, and learn constantly in 2017. And I encourage you to share your favorite ‘A-ha!’ moments in the comments section below.

Enjoy this post? Share with your friends and colleagues: