Putting A/B testing to the test with an integrated optimization program
Doubling revenue 3 years running for DMV.org
|Optimized conversion:||Revenue per visitor|
|Solution:||Integrated optimization program|
|Case study:||DMV Case Study PDF|
Going to your state’s Department of Motor Vehicles can be confusing, complicated and downright frustrating. That’s where DMV.org helps, by providing useful information to simplify the DMV experience: More than 200 million visitors a year look up information on how to register a car, get a driver’s license, learn more about speeding tickets and so on.
“We empower people in their DMV experience by putting them in control,” says Shane Hale, director of site optimization and conversion at DMV.org. This non-government content-driven site provides information about motor vehicle tasks in all 50 states and provides a simplified experience with the DMV across the country.
The thousands of pages of content customized for drivers in each state were fertile ground for a rapid testing optimization program.
With Widerfunnel’s support, the company has built a world-class optimization program that has consistently lifted their revenue. In fact, for three years running, the program has doubled their revenue on their highest revenue products compounded each year. At the same time, we’ve been able to build a framework of customer insights that continue to deliver top results.
Here’s how we did it.
The business need: Increased revenue per visitor
The DMV.org website earns revenue through performance-based advertising on their content pages. When visitors purchase a product like car insurance through DMV.org’s partners, the partner pays DMV.org a referral fee.
For example, on one of the license renewal information pages, a banner within the content offers visitors an opportunity to check car insurance rates. See below.
Because DMV.org is only paid when there’s a transaction, they had an opportunity to use conversion optimization to directly lift their revenue per visitor – and greatly impact their business.
The company’s wide breadth of information and high monthly visitor volume meant that there was tons of potential for optimization. But, the company also knew there were risks if their conversion optimization program wasn’t built the right way.
Challenges faced: Limited staff, unreliable experiment results
DMV.org faced several challenges when they reached out to WiderFunnel. They only had a one-person optimization team.
Their A/B testing relied on a cobbled-together technology stack using Optimizely combined with an advertising server and Adobe SiteCatalyst for revenue reporting, leading to mismatched data and unreliable results.
Most challenging of all, however, was a lack of conversion strategy. The team used what Hale described as a “hunt and peck system” that didn’t create momentum, useful insights, or sustained revenue lift from their efforts.
“We knew there were inefficiencies, but didn’t know how far off we were or what to do about it. We didn’t know if we had good findings or how to make decisions. Too many times we made decisions based on emotion.”
DMV.org needed a better program design and more expertise added to the team to drive consistent optimization results.
The solution: A best-in-class optimization program
WiderFunnel did more than just run a few A/B tests for DMV.org: The teams worked together to develop an integrated long-term optimization program that consistently delivers both revenue lift and customer insights.
The most advanced organizations view optimization as an ongoing strategy. It’s not a one-time project and it doesn’t have an end point where the company’s experiences are “optimized.” WiderFunnel’s Infinity Optimization Process shows how this works for companies like DMV.org.
The Infinity Optimization Process
The process cycles between two distinct phases: Explore and Validate. The two phases support each other, and need each other to succeed. See the next page for an illustration of the process.
Explore is expansive, where the source of new ideas is explored. Validate is reductive; ideas are A/B tested to prove their value. The nucleus in the middle, where the two phases meet, is where Growth and Insights are revealed.
In the Explore phase, the team gathers insights about the business and customers through many different sources: stakeholder interviews, user testing, web analytics and other voice-of-customer analyses.
In this phase, the team also considers publicly-known persuasion principles and internal patterns within the test archive. This is an expansive phase, where all options are considered, whether they are minor usability tweaks or dramatic concepts that challenge the business.
All that data-gathering is centered around WiderFunnel’s framework for understanding the customer’s conversion barriers and persuasion opportunities – the LIFT Model™.
The LIFT Model shows the six conversion factors that are impacting every organization’s conversion rates. It encourages optimization experts to see their marketing experiences through the eyes of their prospects.
Once the team has a good basis for understanding the context and customer, the process moves into the Validate phase. Here, the optimization team designs experiments to find out which of the insights apply to the company’s business context.
The team prioritizes experiments using the PIE framework, develops powerful hypotheses, creates the experimental design to maximize insight-validation, and runs full-service A/B tests.
The Validate phase is a rapid A/B testing cycle where new inputs from Explore are fed into the experiment plans.
The most important final step in the cycle, the results analysis, is where the live experiment data shows how the team’s predictions played out in real life. New insights from these experiments are fed into following experiment sprints and back into the Explore phase.
For DMV.org, the Explore and Validate cycles have produced an ongoing series of profitable and interesting insights. Those insights are what WiderFunnel calls ‘A-ha!’ Moments.
The following pages show just a few of the many A/B tests that have delivered ‘A-ha!’ Moments for DMV.org.
The result: Two years of doubled revenue – and heading for a third
The work DMV.org and Widerfunnel have done together has had amazing results. By supplementing the internal team with WiderFunnel’s full service partnership, DMV.org is now running 500 controlled web and mobile experiments per year, spread across the internal and WiderFunnel teams.
A/B Test: How persuasive is relevance?
Most insurance companies assume their rates are the most important thing to customers. It should be no surprise, then, that most ads reference rates. Who can forget the famous Geico ad tagline? You know, the one about saving 15%.
One of the benefits of DMV.org’s service, which offers multiple insurance providers, is comparing quotes to find the best deal. The insurance banner with this message looked like this:
In testing many different iterations, we discovered that the rates are important, but rate isn’t the only important factor. The relevance of the message is also important.
We constructed an A/B test where one of the variations referenced the visitor’s intent on the page. On an informational page about car registration, we added a lead-in message saying “Registering a car?”, then followed with “It’s a good time to check car insurance rates.”
That change improved the relevance of the message and produced a 14% revenue lift across all content pages with this banner.
Insight: Boosting relevance can produce large revenue lift.
A/B Test: Does regional personalization increase relevance?
If referencing the visitor’s page intent creates a more relevant experience and higher revenue, then what if we pushed relevance even further? Widerfunnel has tested personalized experiences in many different scenarios and found the results to be variable. We were curious to see whether geographic personalization would work in this situation.
In one experiment, we discovered that adding a line under the zip code field saying “Get quotes from the top providers in [State]” lifted revenue by 3.9%.
If such a simple change can lead to that much of a revenue lift, what would happen if we added geographic relevance even more dramatically?
In a follow-up test, we changed the headline,comparing one saying that “1,972,984 Americans” had used the site to research auto insurance rates and another saying that “372,751 Californians” (for visitors from California) had researched rates.
In this case, the result showed the state-specific (e.g. “Californians”) headline decreased revenue by 5.4%!
But, there’s more to this story. After digging into the results at a state-specific level, we discovered significant variance in which states responded better to state-specific messaging. Washington state, at the high end, showed a 67.3% revenue lift when their state was mentioned, yet Oregon (right next door!), showed a 56.2% decrease! See the chart to the right.
These comparisons between states are not statistically significant yet, but this results analysis raises some interesting new questions that can lead to further experiment hypotheses.
The geographic personalization provided other insights as well. We ran another test including the word “please” in the field label and found that there was a geographic association with response: Some states responded better to the label saying “Please Enter Your Zip” than others.
Insight: Geography influences people’s message perception. (Also, are some states more polite than others?)
A/B Test: Things your designer would never recommend
We also tested the impact of changing images, focusing on a mini-banner asking visitors to enter their ZIP codes to get insurance rate estimates.
Widerfunnel and DMV.org ran tests on various images on the banner, including an animated GIF of a fender-bender, a photo of a dog driving a car, a photo of a sad-looking driver, and a banner that had no image but was shaped like a car.
The testing team, including the designer responsible for creating the variations, weren’t sure about that last variation, the car-shaped one. The design was out of left field and some thought it was a waste of traffic to test it.
They were wrong. The car-shaped banner increased the product click through rate by an amazing 89.6% (compared with less than 10% ranges for the other options), which led to a revenue per visitor increase of 74.92%.
Insight: Reserve a portion of your variations for “left field” variations — you might be surprised.
A/B Test: Does a flashy page get results?
On the insurance results pages, we made an effort to reduce noise. Traditional insurance results pages have logos on the left with bullet points differentiating their services. Through our testing, we learned that eliminating the noise worked.
People understand the brands and don’t need to be oversold. We tested a noisy version against an optimized, less noisy page. The optimized page won by 40%.
Insight: Reducing content in a highly branded category can increase conversion rates.
As a result of these tests and others, DMV.org has doubled revenue two years in a row, and a single A/B test — the car-shaped mini-banner — will likely make it three years in a row. But, for many people, there’s an outstanding question about conversion optimization that we wanted to answer.
Do the results stick?
Our long-term working relationship has made it possible to determine whether these results are sustainable. At the six-month mark, DMV.org and Widerfunnel calculated a cumulative multiplied 32.8% lift in revenue. This impressive lift total raised the question: Are these results sticky?
To test this question, the Widerfunnel team used DMV.org’s original control page from before the first test and A/B tested it against the latest winning variation.
In other words, the numbers held, proving that these improvements were sustainable.
This rigorous verification proves that the revenue lift from our A/B testing is not temporary. In fact, the results are, so far, permanent.
Not only is DMV.org getting insights into what works, it’s building a long-term conversion strategy that it will continue to refine and develop in the coming years.
It isn’t just the tests themselves that have made this partnership a success. DMV.org wanted Widerfunnel to be an extension of its own company, and that’s what happened — we’re an integrated team that shares its knowledge so DMV.org can build its own expertise and improve its internal processes for future testing.