6 key insights from the “State of Experimentation Maturity 2018” research report

The optimization industry has transformed over the past decade.

Today, leading companies understand that experimentation is much bigger than A/B split testing on a landing page.

The C-Suite at organizations like Amazon, Netflix, Google, Walmart, and Microsoft are leveraging experimentation as a foundational growth strategy, testing across the customer journey. And other companies are rapidly following suit to avoid becoming obsolete.

But what actually characterizes the organizations that are more successful at experimentation? What are they doing differently? What are they doing better? What are they most focused on?

We set out to attach data to assumptions about “experimentation maturity” with a comprehensive research report. Together, with our partners at Optimizely, we surveyed marketers, product managers, and growth strategists at some of North America’s leading brands like Nike, United Airlines, Showtime, American Express, Hotwire.com, MailChimp and many more.

We wanted to know:

• What are the elements of a mature experimentation program?
• What is the ideal approach for experimentation success?
• What is the experimentation status quo among more mature brands?

And the findings are eye-opening.

In this blog post, you’ll find an overview of 6 key findings from the “State of Experimentation Maturity 2018” research report. And an analysis of the impact of these key insights for organizations that are looking to scale their experimentation programs.

Get the full research report for more in-depth benchmarking data to help you build a business case for scaling experimentation from a departmental initiative to an organizational strategy.

How we analyzed experimentation maturity

For this report, we targeted key individuals at leading North American brands that can test at scale. Which means, on average, respondents worked for companies with websites that obtain 100,000+ unique monthly visitors.

”This target audience was key in order to get an accurate picture of the State of Experimentation from an organizational perspective. We wanted to speak to individuals who weren’t hampered by lower website traffic – who could experiment at scale – in order to better understand the cultural and structural factors that characterize the most successful experimentation programs.”

-Natasha Wahid

Respondents to this report include professionals with Manager-level-and-above titles in Product, Optimization, E-commerce, Analytics, from across a wide variety of industries: Travel, Education, Saas, Retail, and others.

The survey had 26 questions in total – the goal was to assess maturity from a quantitative perspective. Questions included, “How many experiments have you run in the past 12 months?” and “are there clear success metrics for your experimentation program?”

We looked for data points that correlate with success for the most mature experimentation programs, in order to provide benchmarking data.

To supplement the quantitative findings, we also conducted several in-depth interviews with key respondents to find out what these data points mean in real life. What insights have these individuals learned? What challenges are they facing? What efforts are they are spearheading now?

Our Strategy and Data Analysis teams identified five levels of experimentation maturity:

The 5 Levels of Experimentation Maturity

Informed by the data in this report and by the stages of transformation we have observed with our clients, WiderFunnel’s five levels of experimentation maturity are:

Level 1: Initiating

Organizations at this stage are just getting started. An Optimization Champion is working to get initial wins to prove the value of an experimentation program.

Level 2: Building

In this stage, an organization is bought-in on the value of experimentation and an Optimization Champion or team is establishing process and building the infrastructure to scale the program.

Level 3: Collaborating

Organizations at this stage are expanding the experimentation program and collaborating across teams. Finalizing a communications plan and overall protocol for the program is a priority here.

Level 4: Scaling

Experimentation is a core strategy for these organizations. Standards are in place and success metrics are aligned with overall business goals, enabling testing at scale.

Level 5: Driving

The highest level of maturity. Experimentation is the organization’s growth and product strategy. The Amazons, Netflixs, and Booking.coms are here.

These five levels are based on WiderFunnel’s pillars of experimentation maturity: Process, Accountability, Culture, Expertise, and Technology. Each of these pillars is important; they work in synergy with each other and they each need to be developed within an organization.

The Pillars of Experimentation Maturity

The following pillars of experimentation maturity are based on WiderFunnel’s decade of experience helping organizations across industries succeed at experimentation and scale their programs. They are:

Process & Accountability

The most mature organizations keep process and accountability at the core of their experimentation strategy, fuelling how experiments are developed, and results are analyzed, understood, and leveraged.

This pillar includes an organization’s experimentation protocol and methodology, process for ideation and prioritization, experiment design, and measurement of success.

Culture

Culture is crucial when defining experimentation maturity: Does your organization celebrate testing and learning? Are people encouraged to try (and fail) and try again?

This pillar includes organizational buy-in for experimentation, program support from the C-level, and cross-team participation in an experimentation program.

Expertise

An experimentation program needs expertise and resources. The amount of time and full-time team members dedicated to experimentation is reflective of an organization’s maturity.

This pillar includes people and skill sets: strategists, analysts, designers, developers, project managers, product owners, third-party partners, as well as hours dedicated to experimentation.

Tools & Technology

Experimentation maturity requires a well-rounded technology stack. Experimentation and personalization tools, visitor engagement tools, customer data tools, project management tools.

Mature organizations have the right tools in place to ensure they can develop the best possible hypotheses and have reliable data.

6 key insights from the research report

The survey questions that informed this report were designed to address each of these pillars. Organizations that participated in the survey were grouped into one of the five maturity levels based on their responses.

The following section explores 6 key insights from the research report. Get the full 45-page report and access the complete findings, which include:

• Reported conversion rate changes, experiment win-rates, and experiment velocity for each level of maturity
• How the most mature organizations are staffing for experimentation and optimization, and how their experimentation programs are structured
• Learnings and suggestions from strategic experts for how to maintain and scale your experimentation strategy

1. Most respondents are still only scratching the surface when it comes to experimentation.

Experimentation at most organizations is still largely in its infancy, with zero percent of respondents achieving the highest level of maturity in this report.

Half (50%) of respondents are still in the first two stages of maturity, and 33% are in the third stage. None of the respondents in this report have reached the highest level of maturity: “Driving”.

2. Most organizations plan to pick up experimentation speed (literally) in the next 12 months, indicating the strategy is gaining momentum.

While there is room to mature, experimentation is clearly gaining traction for many organizations as a worthwhile strategy. To maintain momentum, most optimizers represented in this report are striving to increase their testing velocity.

”A large percentage of experiments don’t win. Improving throughput (or number of tests per property per month / week) will ultimately generate more success due to the law of large numbers.”

– Hudson Arnold, Senior Strategy Consultant at Optimizely

In fact, the majority of both Small and Medium Enterprises (52%) and Large Enterprises (64%) plan to increase experiment velocity in the next 12 months. Not a single respondent to this survey is planning to decrease experiment velocity in the next year.

Key Takeaway

There are several factors that influence an organization’s ability to experiment at a high velocity, such as:

• Sample size – the number of people in a given test will affect how long that test takes to complete (at statistical confidence)
• Resources – how efficient are your tools and how proficient is your optimization team?
• Workflow – how easy is it to proceed from one step to the next when designing and launching an experiment?

”Determining when to call an experiment complete is not just about sample size. You also need to consider your existing conversion rate and the elasticity of visitor behavior on your primary conversion goal in relation to sample size.”

James Flory, Optimization Strategist of WiderFunnel

3. At 63% of more mature organizations, Executive teams are involved in metric-setting for experimentation, pointing to Senior-level buy-in.

Success metrics and key performance indicators ensure optimizers are held accountable for contributing to business goals and driving actual growth, ensuring the longevity of the strategy.

And when the Executive team is involved in the creation of these metrics, experimentation is linked to the goals they care about, and the experimentation program has visibility at the highest level of an organization.

In fact, Executive teams at 63% of more mature organizations are involved in setting success metrics for experimentation, showing an investment in the outcomes and, even more so, in the strategy’s long-term importance.

But when we analyzed the entire respondent pool, we discovered that less than 50% of respondents have metrics set at the top level.

Without buy-in and alignment at the top-level, optimization champions may experience roadblocks in sharing results and learnings, obtaining resources, and scaling experimentation efforts across the organization.

Key Takeaway

Involve your Executive team in the creation of success metrics for your experimentation program. This will allow you to:

1. Ensure experimentation is linked to the goals your Executives care about, and
2. Ensure your program has visibility at the highest level of your organization.

”If your Executive team doesn’t have any input on your experimentation program’s key metrics, you might be running experiments on goals that they really don’t care about – that aren’t really important to the business.”

– Nick So, Director of Strategy at WiderFunnel

4. Structuring your experimentation program in a combination organizational model may pave a path to experimentation maturity.

Experimentation programs often fall into one of the following organizational structures:

• Decentralized (All-owned testing): Optimizers exist in different teams across the organization. Each strategizes and executes experiments according to their own KPIs, as they are positioned to reach larger organizational goals. This model does not have the oversight of a central team.
• Centralized (Center of Excellence): Often in companies where individual teams have their own sites or domains. A central body that owns experimentation across the business, encouraging the growth of skills and the expansion of the program.
• A Combination Model: When there is shared ownership of digital initiatives, a central body works with all relevant stakeholders to prioritize and plan for experimentation. But individual teams have the ownership to execute tests.

There are pros and cons to each experimentation program structure, and optimization champions will need to figure out what will work best for their organization and their unique culture.

While a decentralized program may enable speed of testing, without proper standards and central oversight, there is a risk that teams will clash, cannibalize each other’s goals, and pull in different directions rather than pushing the organization in a unified direction.

A central body will most likely have the holistic customer journey in mind, and can prioritize and deliver experimentation campaigns that focus on the most important parts of the business. But it’s unwise to assume a central body will have the necessary depth of expertise to experiment on certain parts of the business.

A combination model attempts to combine the best of both worlds and is favored by the most mature organizations, with 58% using a combination model.

…and it clearly works. We found a correlation between higher experiment velocity and a combination organizational structure.

In the combination model, there is shared ownership of digital initiatives – a central body works with all relevant stakeholders to prioritize and plan for experimentation. But individual teams are able to own and execute tests.

52% of organizations that are running experiments at a high velocity are using a combination model, showing the agility of the organizational structure.

“While a centralized experimentation team like a Center of Excellence may be the ‘hallmark’ of a mature experimentation program, it is not surprising that a combination of both team structures is most efficient.

It truly speaks to the collaboration required in achieving organizational experimentation maturity. No matter how mature the central team is, if the organization is lacking the buy-in from outside teams and business units, that experimentation team cannot reach it’s full potential.

While the centralized team should act as the core of the experimentation, part of that is to disseminate the mindset across the organization for alignment. It should not be the sole isolated contributor to an entire organization’s experimentation activity,” explains Nick So.

5. More mature organizations overwhelmingly tend to seriously consider any experiment idea that is backed by evidence.

95% of organizations at the highest maturity level in this report – Scaling – “Agree” or “Strongly Agree” that all experiment ideas are seriously considered as long as there is evidence to support them.

This means that more people can submit potential test ideas for consideration as long as they can provide solid data-backed reasoning.

It also means that more people are accessing data, including experimentation results, in order to form solid hypotheses to fuel their experimentation programs.

In contrast, ⅓ (32%) of overall respondents are neutral or disagree.

These organizations may be missing out on winning experiment ideas. Because no one person or single team at an organization will have in-depth expertise at every touchpoint within the customer journey.

Mature experimentation organizations understand that every member of a company may have a brilliant idea to test to improve the customer experience.

“Every department should be encouraged to submit ideas for experimentation. But this should only be done when a company is also confident it can complete the feedback loop and provide explanation as to the acceptance or rejection of every single idea.

An incomplete feedback loop – where people’s ideas get lost in a black hole – is one of the most detrimental things that can happen to an experimentation culture.

Until a feedback loop can be established, it is better for a more isolated testing team to prove the value of the program, without the stressors caused by other parts of the organization getting involved.” – Mike St Laurent, Senior Optimization Strategist at WiderFunnel

The challenge for these organizations as they attempt to scale experimentation is to create a system that educates everyone about the program, that requires evidence with each idea submission, and that provides a closed feedback loop for the consideration of ideas.

6. Across the board, organizations are focused on resourcing for Web Development and QA skills.

When building a team for optimization, organizations at every maturity level are highly focused on Web Development and QA. These skill sets are important to ensure experiment variations can be coded and tested for quality pre-launch, and that winning variations can be hard-coded quickly.

The other skill sets – Design, Experimentation Strategy, Data Science & Analytics, and Project Management – are a much lower priority for all survey respondents. Resourcing for Design and Experimentation Strategy is only slightly preferred to resourcing for Data Science and Project Management.

Mature organizations, however, are also highly focused on hiring Experimentation Strategists.

While Web Development and QA skills are a priority for every organization surveyed, organizations at the “Scaling” maturity level are equally focused on Experimentation Strategy.

Respondents in this category reported double the average number of team members involved in “Experimentation Strategy” relative to any other maturity level.

This focus on strategy indicates that mature organizations are hiring experts who have a combination of strong product management basics – the ability to define requirements, collaborate with teams and stakeholders – as well as experimentation and analytical rigor – the ability to interpret test results properly, estimate test durations, and own the science of experimentation.

”Getting the right team in place to do optimization is essential. I think a lot of it is about the ability to evangelize experimentation across the organization. To do it in a way that isn’t in your face, and is very smart. To do it in an emotionally intelligent way. When I was building my team, emotional intelligence was an extremely important consideration – it wasn’t just about technical capabilities and skills. And that has contributed a lot to the success of where we are today.”

– Lauren Schuman, Director of Growth at MailChimp

Leading brands see experimentation as a strategy to enable business transformation. However, the research clearly shows that most companies are still in the early stages of maturity.

The most mature organizations are making huge gains in terms of getting organizational buy-in and creating a culture of experimentation. These organizations have adopted a test and learn business mindset.

Instead of operating at the gut-level, they are building data-driven strategies with their experimentation at the heart of every new product or feature; their Executive teams are embracing the you-should-test-that philosophy and applying it across the entire customer journey. And this is leading to more refined, more delightful customer experiences.

But with the competition this stiff, the real question to ask is: How does your organization compare?

Get access to the full research report, below, and find out!