Experimentation fuels growth. Organizational culture supports experimentation.
Experimentation is a core strategy for product development and growth at organizations that are winning in the market.
In a report on the Insights-Driven Business, Forrester demonstrated that businesses with closed-loop learning processes at their core are growing at least eight times faster than the global GDP. These learning processes have experimentation at their center.
Leading companies like Netflix, Uber, Amazon, Airbnb, Microsoft are quick to share that they are fuelled by experimentation. These are what we would call mature organizations—they leverage experimentation to generate continuous insights and growth that impact their bottom-line.
Pursuing experimentation maturity
At WiderFunnel, we have been working with brands to build and scale insight-generating experimentation programs for over 12 years. In doing that work, we have identified five phases of experimentation maturity that organizations progress through.
Level 1: Initiating Organizations at this stage are just getting started. An Experimentation Champion is working to get initial wins to prove the value of an experimentation program.
Level 2: Building In this stage, an organization is bought-in on the value of experimentation and an Experimentation Champion or team is establishing process and building the infrastructure to scale the program.
Level 3: Collaborating Organizations at this stage are expanding the experimentation program and collaborating across teams. Finalizing a communications plan and overall protocol for the program is a priority here.
Level 4: Scaling Experimentation is a core strategy for these organizations. Standards are in place and success metrics are aligned with overall business goals, enabling testing at scale.
Level 5: Driving The highest level of maturity. Experimentation is the organization’s growth and product strategy. The Amazon’s, Netflix’s, and Booking.com’s are here.
The organizational culture component
While there are multiple pillars of a mature experimentation organization—such as a powerful technology foundation and clear objectives for the program—developing a culture of experimentation is essential. In order to move from one phase into the next, you must foster an organizational culture that embraces testing and learning.
Ultimately, a culture of experimentation is a factor of effective communication. If you can’t get this piece right, your organization will never reach the highest level of maturity.
A hypothetical example
Imagine you are an Experimentation Champion trying to build a testing program from scratch. You have a vision for experimentation; you believe in its value and are excited to get moving. You are a one-person show, but you are convinced you can get the rest of your company on board.
To do this, you decide to get everyone involved in the experimentation program at the outset. You open the program up to the whole organization. You start sourcing ideas from everyone and everywhere.
But there’s a problem: Not everyone understands experimentation. People are contributing ideas without thinking about them, without thinking about the greater context. Soon, you are buried under ideas you can’t actually execute on.
And the experimentation program becomes a joke; a dumping ground where ideas go to die and nothing seems to be getting done. The teams around you lose faith and gradually stop submitting ideas. Eventually, experimentation becomes…
“Something we tried once. It doesn’t work for us.”
We see this story unfold all too often. But this doesn’t have to be your story. You, as an intrapreneur, can foster a culture of experimentation and avoid this crash and burn scenario.
Inspire. Educate. Inform.
You can do this by leveraging three essential actions in your communication: Inspire, Educate, and Inform.
Inspire means creating the spark. Inspiration occurs when you create a moment of clarity and awareness of new possibilities, as well as a desire to take action—to get involved.
Inspiration […] involves a moment of clarity and awareness of new possibilities. This moment of clarity is often vivid, and can take the form of a grand vision, or a “seeing” of something one has not seen before (but that was probably always there). Finally, inspiration involves approach motivation, in which the individual strives to transmit, express, or actualize a new idea or vision. [It] involves both being inspired by something and acting on that inspiration.— “Why Inspiration Matters“, published in the Harvard Business Review
Educate means training. This action involves training, by instruction and/or supervised practice, in a particular skill. In this case, how to do experimentation.
Inform is closing the loop. Inform means actually communicating a message or making something known.
In the journey to a mature culture of experimentation, these three actions will need to be at play at different levels.
When your organization is just getting started with experimentation, testing is likely owned by a single, core team. Inspire, Inform, and Educate must be at work in this core team before your organization can move into the next phase.
To scale, your organization will need to empower supporting teams to participate in the experimentation program. The ultimate goal being an organization where every single person has an experimentation mindset, from your CEO to Lead Engineer, to the customer support heroes who pick up the phone everyday.
To get here, you will have to continuously Inspire, Educate, and Inform, to drive organizational change and foster the experimentation mindset. But what does that actually look like?
Just getting started with experimentation
In the early stages of maturity, a core team is often responsible for running optimization experiments. They are likely focused on getting initial buy-in for testing and building momentum around positive results.
One or a few team members should also be focused on Inspiring, Educating, and Informing necessary stakeholders, to lay the culture of experimentation foundation.
Inspiring your core team
As an Experimentation Champion, your first priority should be to recruit a core experimentation team. This team should include an Executive Sponsor (if that isn’t you), and individuals or partners who can execute experiments: Design, Engineering, Data Science, Experimentation Strategy, etc.
You may have these resources in-house, or you may decide to bring in an enabling partner to augment your capabilities. Either way, your core team should help you 1) develop and prioritize experiments, 2) execute experiments, and 3) socialize experiment results.
You will need to inspire the members of your core team to motivate them to get involved and stay involved in experimentation. At the outset, this means tailoring your message to each individual and showing them the new possibilities of testing. You should constantly ask yourself:
- Who am I speaking to?
- What do they care about?
A real-world example
One of our clients is a technology startup. Several months ago, the Head of Demand Generation decided to implement an experimentation program. She knew she needed to start by recruiting a small core experimentation team.
She began with an Executive Sponsor—the company’s VP of Revenue. This VP has an allstar sales background, but wasn’t familiar with the concepts of “experimentation” and “conversion optimization”. She needed to show him new possibilities that would matter to him.
The company had just finished a website redesign. In this context, the Champion worked carefully to explain to her VP that, while the project had succeeded from a brand and aesthetic angle, there were still potential points of friction in the user experience.
She pointed out that the company was spending substantial money to funnel traffic to the redesigned website, and emphasized the missed opportunity of not addressing these potential barriers to conversion—the opportunity to increase their primary metric by 2%, 5%, 10%.
She spoke to the VP in financial terms that mattered to him, and showed him the financial possibilities around marketing experimentation. And he got on board because he was inspired.
Educating within a core team
At this stage of experimentation maturity, Educate and Inform can often be done informally.
Your core team should have the skills to do experimentation, but they may not have complete understanding around why and how to do it. Design may understand UX best practices, but they may be wary of marketing experiments that could challenge brand standards. Engineering may be highly focused on product development, and lack the front-end development experience needed to develop marketing experiment variations.
Your best course of action in this case is to involve your Designer(s) and Engineer(s) in the conversation as early as possible. Remember, these are members of your core team. As such, they should be a part the experimentation conversation from start to finish.
This is education via supervised practice: by doing. As the Experimentation Champion, you should be guiding the conversation around overall objectives and experimentation frameworks. You should be educating the other members of your core team.
If you yourself are unsure about the in’s and out’s of testing, it may be a good idea to bring in an experimentation partner. With our technology client, the Champion knew she was missing critical pieces of a core team, including strategic support and dedicated resources. Which is why she brought in WiderFunnel as an enabling partner. In that role, we are able to work with her to transfer knowledge around experimentation to the members of her core team.
Informing within a core team
When it comes to Inform, you must make sure that you are closing the loop with each stakeholder on your core experimentation team.
This means informing your team members when an experiment is launched and informing them when and why it is completed. It means including them in the results analysis conversation and in determining next steps.
If someone is involved in an experiment, you must keep them informed, particularly regarding the impact of that experiment. Whether this is via email, Slack, or simply a face-to-face conversation, the importance of closing the loop cannot be overstated. Because nothing is quite as motivating as seeing the bottom-line impact of your work. And nothing is quite as de-motivating as contributing to a project and not knowing the results of your contribution.
One of the most important things to note at this stage is not to overreach. Focus on recruiting the core team and resources needed to get your experimentation program rolling. Work to Inspire, Educate, and Inform these key people—Engineering and Design, your Executive sponsor, and your Executive team.
As you scale the experimentation program and begin to empower supporting teams, your core team will need to Inspire, Educate, and Inform these supporting teams to get them up and running.
Of course, one person can’t shoulder Inspiring, Educating, and Informing for the entire organization. To support scale, you will want to implement systems that help to automate the actions of Inspire, Educate and Inform.
Building momentum for experimentation and driving organizational change
So what does that look like? Let’s look at a slightly more mature organization.
Another partner of ours is a large, digitally mature financial services company. When we partnered with this company, there was already Executive-level buy-in for testing, as well as a general understanding of the value of experimentation.
The core experimentation team had been assembled. It consisted of an Experimentation Champion, an Executive Sponsor, and WiderFunnel as an enabling partner. The function of this core team was to enable supporting teams (rather than to execute experiments). As an organization matures, this is often the role that a core experimentation team moves into—a facilitating, enabling role rather than an executing role.
At this particular organization, the core experimentation team was trying to enable eight different product marketing teams to develop and launch digital experiments.
Taking advantage of opportunities to Inspire your organization
In this case, the Experimentation Champion had taken advantage of an opportunity to inspire the larger organization.
One of the highest visibility product teams at this company had just gone through a page redesign, which was performing terribly. To address this, the Champion brought in WiderFunnel to analyze the redesigned page, identify potential barriers to conversion, design a variation, and launch an experiment to try to improve page performance.
The core team was very confident that the experiment would win because the redesign was performing so terribly. There was a lot of potential. And they were right—the variation performed much better than the redesign. Because this was such a high-visibility product, the whole organization was watching.
The lesson here? Be opportunistic about promoting the experimentation mindset. The other seven product teams saw, first-hand, the new possibilities associated with experimentation. And they were chomping at the bit to get started; to get involved.
But first things first.
At this stage, documentation becomes critical. When one small team owns and operates experimentation, you can get away with little to no documentation. But as you expand into supporting teams, centralized documents, processes, and standards become necessary to enable proper knowledge transfer. Documentation is one of the systems that helps to ‘automate’ the actions of Inspire, Educate, and Inform.
Enabling Inspiration, Education, and Informing at scale
To do this, the core team at this organization decided to leverage two primary activities: workshops and documentation.
Workshops became the primary vehicle for education and continued inspiration; documentation became the primary guide for how and whom to inform.
While the Champion in our previous example was educating her core team informally, via practical instruction, this organization needed to take a more formal approach to educate eight separate teams.
The core team needed to transfer knowledge around how to run experiments, but perhaps more importantly, around how to think about experimentation. The supporting teams had a very narrow view of ‘experimentation’. It was seen as a UX tactic—testing small tweaks to improve a particular conversion metric.
We wanted to educate these teams on the true potential of experimentation—how to use it to get real answers to real questions. To do this, we designed a series of workshops that walked these teams through various questions that they could and should be asking when developing experiment hypotheses.
There were questions that asked teams to refocus on their overall objectives and contextualize any experiment ideas within their broader goals.
There were questions that asked teams to refocus on their website visitors and customers and develop ideas based on qualitative and quantitative data: who their customers are and how they are using the website.
The workshops were also designed to lay a foundation for collaboration, asking teams to consider the experiments other teams were running and whether these insights might be relevant to their products and digital experiences.
These workshops were an interactive learning opportunity between the core team and the supporting teams. And this education was also inspirational: In asking new questions, the different team leads were able to envision entirely new possibilities for how to better develop and position their products leveraging experimentation. They were able to get excited about the possibilities.
Alongside these workshops, the core team was working to document a communications plan. The plan would have two main components:
- Clarifying roles and responsibilities as the experimentation program scales, and
- Clarifying who needs to be informed and how
To clarify roles and responsibilities, we recommend leveraging RACI, or a similar model. This model helps you map out who owns which piece of the overall task.
R = Responsible: The person who does the work to achieve the task.
A = Accountable: The person who is accountable for the correct and thorough completion of the task.
C = Consulted: The people who provide information for the project and with whom there is two-way communication.
I = Informed: The people kept informed of progress and with whom there is one-way communication.
Along with clarifying roles and responsibilities, you need a documented communications plan, which should:
- Identify the information that needs to be communicated
- List the methods of communication (formal and informal) and how they’ll be utilized
- Determine the line of communication: Who communicates to whom?
- Be intentional with timing
MailChimp is an example of an organization that prioritizes communication, and they have seen a lot of success in scaling their experimentation program. I first spoke to the Experimentation Champion there a year ago—she was just getting the program off the ground in the Marketing team. Today, MailChimp is testing on their Marketing site, on 3 of 9 of their product domains, as well as within their technical content team.
The Champion I’m referring to is Lauren Schuman, now Senior Director of Product Insights & Growth. Lauren explained that documentation and communication were a priority from the very beginning at MailChimp.
We documented every single step of the workflow and did a RACI Model associated with it—so that we were very clear on who was doing what, who we needed to consult versus inform. We then mapped the actual communication strategy […] And this has been a major contributor to how we’ve scaled and how we’ll be able to scale in the future.—Lauren Schuman, Senior Director of Product Insights & Growth at MailChimp
Recently, Lauren shared that she now refers to her internal communications plan as a “marketing plan”, adopting a different mentality. She is marketing the experimentation mindset internally, identifying segments, channels, and relevant messages. The intention is to drive enthusiasm, affinity, and adoption.
Getting your organization psyched about experimentation
Inspire, Educate, Inform. These three actions are tools you must use as you scale your experimentation program. You should determine when to create systems that enable these actions at each stage of maturity.
When you assemble your core experimentation team—as the Champion, you have to make sure that you are Inspiring, Educating, and Informing your key stakeholders. Whether this is happening informally or formally, your core team should be excited about experimentation, they should understand it and its value, and the information loop should be closed before you attempt to expand the program into supporting teams.
As you scale, build systems that will help you automate these actions as you pursue the ultimate goal: A culture of experimentation that permeates your entire organization.
Keep in mind that your communications strategy, like experimentation, is an iterative process. Don’t expect perfection right away; work with your core team and, eventually, your supporting teams to determine which messages and channels work best. Be willing to evolve your strategy, but stay committed to Inspiring, Educating, and Informing the people around you.
Are you working to shift organizational culture and promote an experimentation mindset? What challenges are you facing? What successes are you seeing? We’d love to hear from you! Leave your thoughts in the comments section below.