A-ha! Isolations turn a losing experiment into a winner

1  | Last updated: June 28th, 2018

Editor’s Note: We just published our 5 favorite ‘A-ha!’ moments from 2016. >> Check ’em out here! <<

Who doesn’t love a story of redemption?

Annie Selke is a multi-faceted company that sells home-ware goods, including home textiles, rugs and luxury linens.

Annie Selke homepage
The Annie Selke homepage.

We worked with the Annie Selke team over the course of a year to improve conversion rates across their site (which includes five different brands).

Part I: The ‘losing’ test

We ran an experiment on product category and search results pages that featured a left-hand filter. We had tested this filter previously and knew that there was some sensitivity around its structure.

We wanted to explore further.

In our first variation, we isolated a single element, changing the tiled ‘Sort by’ functionality to a more standard drop-down format.

The second variation was built on the first, but with it, we collapsed the left-hand filter to show only the top four search options.

Both of these variations lost to the control page.

Part II: The turn-around

We had two losing variations on our hands.

But because this test had been structured using isolations, we were able to walk away with a valuable insight:

When we looked at the results by isolation, variation B (changing the filters), which was built on variation A (changing the sort by function), showed the potential for an approximate 20% lift in conversions.

By creating specific and progressive isolations you can compare them across variations
By creating specific and progressive isolations you can compare them across variations.

We had also collected some compelling heat map data during the previous experiment that showed users pay particular attention to the word ‘color’.

We went back to the drawing board and designed a new experiment based on these insights.

Variation A retained the collapsed left-hand filter from the previous experiment’s variation B. But, we built the new variation on the original control page, eliminating the losing ‘Sort by’ element.

Variation B was almost identical to variation A. Our only adjustment was to change the word ‘palette’ to the word ‘color’ to address visitors’ fixation with the word.

The final variation was built on variation B, but we made the side filter sticky as the user scrolled down the page.

The winning variation (yes, this test saw a clear winner) lifted sales by 23.6%.

A-ha!

Our ‘losing’ experiment revealed an insight that led us to a huge winner, underlining the importance of experiment design.

Michael St Laurent

The original test may look like a loser, but it’s actually a winner. That’s the power of isolations and the reason you shouldn’t always strive for big changes: when you make a lot of changes at once, you don’t know which change is causing the results.

Michael St Laurent, Optimization Strategist

This post is second in a 5-part series. If you missed our first ‘A-ha!’ moment, check out the link below:

A-ha! Personalization goes beyond design

Stay tuned for more of our favorite ‘A-ha!’ moments in the coming days.

Author

Natasha Wahid

Marketing Manager

Discover how your experimentation program stacks up!

Benchmark your experimentation maturity with our new 7-minute maturity assessment and get proven strategies to develop an insight-driving growth machine.

Get started