Back in 1975, Kodak had just developed a digital camera—the first of its kind. But the product was dropped for fear it would threaten Kodak’s photographic film business. At a meeting of the Kodak board, one member exclaimed that nobody will ever want to look at their photos on a screen.
You could argue that Kodak made a choice, in that boardroom, that would ultimately result in the company’s bankruptcy. A choice to stifle innovation for fear of failure.
But as we’ve seen over and over again, to survive and thrive, an organization must balance optimization with innovation, exploitation with exploration. Deborah Wahl, Global CMO at Cadillac, former CMO of McDonald’s explains,
As a CMO, you need to really understand the core balance: What makes money in your company and how? And what’s enough—what’s the minimum level that you need to invest to keep that flywheel going? Then carving [that amount] out and absolutely being relentless about spending the rest on innovation and forward looking. Because your biggest bumps [will come from] category creation.
Innovative exploration, however, does not have to be a gamble. By leveraging experimentation, organizations can test and validate a business idea before leaping. Experimentation allows innovation with confidence, innovation that is customer-centric rather than HiPPO-driven.
“[…] Big data can provide clues only about the past behavior of customers—not about how they will react to bold changes. When it comes to innovation, then, most managers must operate in a world where they lack sufficient data to inform their decisions. Consequently, they often rely on their experience or intuition.
But ideas that are truly innovative—that is, those that can reshape industries—typically go against the grain of executive experience and conventional wisdom. Managers can, however, discover whether a new product or business program will succeed by subjecting it to a rigorous test,“ write Stefan Thomke and Jim Manzi in Harvard Business Review.
And this approach works.
According to The Gartner Survey Analysis: High Performing Organizations Make Testing, Experimentation and Analytics a Data-Driven Marketing Priority, “organizations that significantly outperform their peers are almost twice as likely to prioritize testing and experimentation.”
Which may be why 88% of companies say that digital experimentation will play an important role in keeping businesses competitive in the next three years, according to new research from Optimizely and Sapio Research.
Organizations, both large and small, incumbent and disruptor, are acknowledging the importance of experimentation. Now, the big question is how: How does an organization embed experimentation practices into its culture and its processes?
How do you build a team, and ultimately an organization, that does experimentation well by hiring for the right skills, breaking down silos to spur cross-team creativity, and by looking at what matters most when it comes to building a long-term innovation strategy.
With insights from our own WiderFunnel experts, industry thought-leaders, and experts at top-performing teams, we’ll hear what the pros suggest you can do to overcome roadblocks in the path to innovation via experimentation.
Related reading: “Why experimentation programs fail (and three steps to get back on course)” via GO Group Digital
Benchmarking your experimentation program
Experimentation pioneer, Chris Goward, explains that at a basic level, every organization should have a goal of developing a core competency of experimentation excellence.
“If your [experimentation] program is just starting, your results aren’t what you expect, or experimentation isn’t integrated seamlessly throughout the company, you have work to do,” he says. Goward explains that conducting a gap analysis of your company against the attributes of great experimentation programs is often a good place to start.
“Our research has shown there are five attributes to study: Process, Accountability, Culture, Expertise, and Technology, which we [at WiderFunnel] call the Experimentation PACET.”
Benchmarking your strengths and gaps within these five attributes will require commitment, debate, and collaboration from key stakeholders across your organization. But it is a necessary exercise that will allow you to prioritize the most important constraints and determine how best to alleviate them.
Setting an intention for experimentation
“If you’re not experimenting, you’re falling behind—I agree,” says Matt Wright, Director of Behavioral Science at WiderFunnel. “However, conducting experiments for the sake of experimenting is not a strategy, either. Instead, experimentation should be conducted with purpose and intent.”
For some organizations, the objective is to optimize existing digital products and experiences and increase sales: These goals tend toward optimization and exploitation.
For others, experimentation is a strategy for validating new products and features: An insurance policy aimed at reducing risk. For others still, the primary objective is to uncover and validate insights about the business and its customers that spur innovation.
The best programs balance all of these objectives.
According to Wright, the intention for your experimentation program should be focused on trying to deeply understand and design for the humans behind the numbers in any given experiment.
Related reading: “The business of behavioral change”
Your organization’s highest-level objectives with experimentation should be filtered into metrics and KPIs that hold your experimentation program accountable. Are you achieving your objectives? Are you making an impact?
“Without goal alignment from the very beginning, experimentation will go sideways quickly. You’ll see teams cannibalizing each other, prioritizing their own individual metrics over the big picture,” says Goward.
Recommended next reading
Moving the needle: Strategic metric setting for your experimentation program
Maintaining a customer-centric focus
Experimentation is powerful because it provides a continuous feedback loop wherein your customers can show you the efficacy of your strategies.
“Decision making becomes extremely easy once your experiments start revealing real [customer] data. You’re no longer sitting in the boardroom making decisions based on gut feeling. The conversation shifts to data and hypotheses, and any idea that comes up is suddenly a candidate for an experiment. Everyone realizes that it’s actually easier to do your job if you are running experiments and letting the data guide the way,” says Ralph Chochlac, VP of Product at GraphPad
But maintaining a customer-centric focus requires studying and analyzing the right metrics, to ensure you are continuously delighting your customers.
Imagine pop-up, ad-ridden, and forced-action interstitials that make for a miserable user experience but may signify success in certain metrics such as click through rate, ad viewability, or something similar. Not a great experience, right?
If you’ve got weak metrics that you can easily game, then you may be inaccurately reporting success when, in reality, you’re eroding any goodwill or loyalty your customer has toward you because you’re creating a frictional experience.
Instead, Flory recommends incorporating guardrail and lifetime value metrics into your experimentation program wherever possible to get a clear, honest picture of performance around customer-centric experiences.
Finding and retaining your experimentation experts
Intention is one piece of the puzzle, execution is another. To do experimentation, you need experimentation experts. But these abilities are tough to gauge during the hiring stage, which means attracting the right talent (and properly training them) can be a major obstacle for organizations.
This matters when it comes to big-picture growth. According to Optimizely’s report, How to Win in the Digital Experience Economy, 81% of business leaders say skills in experimentation and optimization are essential for business growth, but employee skill gaps are in the way of progress.
From the report:
“Past experience suggests that developing existing employees or attracting talent with the right skills will be pivotal to making a success of experimentation. 54% of businesses claim that previous experimentation adoption attempts have stalled due to a lack of skilled employees. Fortunately, many organizations seem to be learning their lessons, as 85% are planning to address the skills gap.”
Data from another study on the challenges to online controlled experiments from experts at Microsoft, Google, Linkedin and other top technology companies echoed these findings. The report notes major obstacles in the way of scaling experimentation efforts such as maintaining quality standards, accurate result quantification, and building a culture of experimenters.
It’s clear that most companies have a lot of work to do when it comes to hiring and training to build a team that pivots around constant experimentation.
Addressing gaps: Experimentation skills most organizations seem to be missing
When it comes to experimentation specialists, the best will have competency in a variety of skills, including an understanding of qualitative research, data, statistics, consumer psychology, marketing, and design thinking. They will also have a particular mindset that marries data-driven with creative thinking.
Nick So, WiderFunnel’s VP of Client Services, says that when an organization is looking to launch a successful experimentation program, there are specific qualities to look for in candidates.
“Try to hire the most experienced experimentation specialist as you can find. It’s easy for a program to be led down the wrong path with individuals who may have simply ‘dabbled’ in testing or optimization. It’s easy for anybody to jump in and start running tests, only to find out six to 12 months down the road that the tests were all inconclusive or the primary goal was not appropriate,” he explains.
“True practitioners of experimentation will know that simply running A/B tests is only a tiny part of actually launching an effective experimentation program.”
Aside from hard skills, you should also look for Experimentation Activators.
John Ekman, Founder of Conversionista in Scandinavia, explains, “The goal of experimentation is not to be right, it’s to make a business impact, to power more confident decision making that results in action. To move from concept to action, you have to have experimentation ‘activators’ on your team.”
These “Activators” make things happen. They are the individuals within an organization who can turn thoughts, ideas and concepts into action. They can translate experiment ideas, evidence and results to stakeholders at the executive level and across teams, generating buy-in and excitement as well as enabling individuals to act.
Recommended next reading
7 habits of highly effective conversion rate optimization strategists
Recommended next reading
How to build a high-performance marketing team
Silos: A barrier to experimentation-driven innovation
While hiring the right individuals is essential, connecting those individuals to each other is just as important. Andre Morys, Founder of Germany’s leading optimization company, konversionskraft, put it this way: “Silos are where innovation goes to die. Insights come from ‘two-pizza teams’ tasked to take on a journey from A to Z.”
In fact, 91% of respondents to Optimizely’s report said that a lack of ideas is not the issue when it comes to improving the digital customer experience through experimentation. Rather, it’s silos and organizational structures that are the biggest culprit, that hold these ideas back.
Why organizational silos are a barrier to scaling experimentation
If you want to socialize the experimentation mindset, you need buy-in across all teams. Silos and the mentality of “our team only focuses on this part” is a major barrier to fostering cross-team collaboration and a true spirit of experimentation.
WiderFunnel Director of Experimentation Strategy, Mike St Laurent, explains that compartmentalized teams often come with competing priorities—and that’s a dangerous road to go down.
“Siloing tends to lead to departments that are given budget based on performance for different KPIs. This is fundamentally at odds with a good experimentation and optimization strategy which is trying to get to the greatest group utility. When a single team is trying to experiment in their own interests to protect their team and gain more budget, that can be problematic,” he says.
Krista Seiden, VP Product Marketing & Growth at Quantcast, agrees that the focus on testing has to be company-wide.
“Most companies have a few people who are optimizers by nature, interest, or experience. Some may even have a growth team. But what really moves the dial is when everyone in the company is on board and thinks this way reflexively, with full support from C-level leaders,” she explains.
Experimentation at scale: A roadmap for the enterprise
There’s no one-size-fits-all approach when it comes to establishing a structure for an effective experimentation program. However, one of the secrets to success seems to be shifting the organizational culture to one that values new ideas and celebrates failure.
Fostering a culture that embraces failure
Failure is going to be a large part of any experimentation-driven organization. The average win-rate for most experimentation programs hovers around 25%—which means 75% of your experiments will lose.
“You’ve gotta accept that your business is going to be in many ways an experiment, and it might fail,” Jeff Bezos, founder and CEO of Amazon, told Bloomberg. “…You are going to fail a lot, and it’s OK, and you need to have a culture that supports that,” adds Bezos.
It is critical that your organization set this example at the highest level.
According to Optimizely’s report, 43% of decision makers embrace failure more than less-senior employees. “Embracing failure” is great in theory, but being transparent about your own failures as a business leader can be difficult. Yet it is critical to encouraging your entire organization to try and fail.
“People at the executive level need to set the example and show openness to failure for the sake of innovation. Traditional team building indicates leaders should show vulnerability in order to gain trust. Being open to talk about losing tests or having unexpected results is a good example of this in an experimentation context,” says So.
James Flory agrees. “Encourage failure, as long as you learn from it. Some of the best results come from such conclusive and decisive failures that you have a profound moment of understanding on how to turn that failure around into a win.”
Lauren Schuman, Director of Growth at MailChimp, says, “A big part of successful experimentation is transparency and visibility. Visibility into what you’re doing and learning—and getting people really excited about that.”
Experiment and lead the way
Forrester predicts that the next 10 years will generate an order of magnitude more change than we have seen in the last 10 years. How will you cope? Insights-driven businesses will arm their employees with insight about changing customer habits and competitor weaknesses. Furthermore, they will be able to constantly experiment so they can test scenarios, learn, and adapt.
Building an experimentation-led organization is not easy. But it is a necessary endeavor if you want to continue to innovate and lead in your industry. Insights-driven businesses that are fuelled by experimentation are coming for your market share. If you don’t embrace experimentation as a company, you will fall behind.
Discover how your experimentation program stacks up!
Benchmark your experimentation maturity with our new 7-minute maturity assessment and get proven strategies to develop an insight-driving growth machine.Get started