At Widerfunnel, we love to share experiment-validated insights regularly, both internally and with you, our loyal readers.
We carve out time with our team to discuss methodologies, new research findings, and experiment results in our monthly A-ha! meetings. I hope you’re doing similar knowledge-sharing to facilitate a healthy experimentation culture at your organization.
Not all experiments result in clear, conclusive winners, and that’s ok, as long as we have conclusive insights. With our clients, we shift expectations and focus on the journey with the understanding that insights inevitably lead to winning results over time. Many key unexpected findings surface as we iterate.
“The only real ‘losers’ are experiments where no insights can be uncovered. Some of the most valuable tests are not the big wins but the well-designed experiments that lead to big insights.” – Matt Wright, Director of Behavioral Science at Widerfunnel
Embracing a test-and-learn mentality not only delivers a heartier roadmap and more profound results but also contributes to growing your organization’s experimentation maturity. We routinely suggest leveraging rigorous framework thinking to deliver the best results. If you haven’t read Going from 10 to 100 experiments per year: Building the frame, you will find it useful in helping you increase your experimentation velocity.
Subscribing to a test-and-learn strategy can feel more labor-intensive in the early days; however, you’ll find the payoff over time to be immense. You must have some patience, and it can be a challenge for inexperienced leadership to see the value of this approach. Successful intrapreneurs anticipate that the people and cultural friction often requires more effort to overcome than the technical challenges. We must educate the executive level, as well as the broader organization, of the advantages—the ability to scale your experimentation program to a business-wide strategy being the ultimate end state.
For an informed point of view, listen to our recent Microsoft webinar on creating a culture of experimentation. I highly recommend you take a few minutes out of your day to pick up some tips from one of our favorite clients, Beth Foster, Principal Program Manager, Experimentation at Microsoft. It’s not every day that you can grab some time with an experimentation veteran who develops experimentation and personalization strategies and cultivates experimentation culture across Microsoft.
During the webinar, we received so many great questions from attendees that we wanted to share answers with you compiled by our Widerfunnel team.
If you have other questions you would like our team to answer, email us and we’ll be glad to post the answer.
Question 1: What strategies and tactics can you use to avoid low-value tests vs. more transformational experiments?
Widerfunnel works with our clients to address issues like this by building risk profiles for experimentation programs that aim to create an appropriate mixture of experiments that focus on both optimization (the 2% experiments) and the big leaps (the 2X experiments) depending on the program’s unique context. For more advanced programs, this is especially helpful when going through prioritization exercises and building out quarterly roadmaps.
We regularly conduct training on problem identification, hypotheses creation, and test design. Low-value results are often a symptom of poorly designed experiments that are not focused on real customer problems.
If you would like to learn more, we have built templates and resources to assist with testing, feel free to reach out to and I’ll email you a copy of our format
Question 2: How do you create a testing roadmap when the focus of a company/product may change based on learnings from tests or macro events?
Our team recommends building flexible quarterly roadmaps with a high-level concept and general testing areas to adjust as we move through a quarter. Our experimentation management too, Liftmap, maintains a dynamic, prioritized list of all experiment ideas and areas to optimize. As new data is collected and new ideas entered, they are all scored based on the PIE framework so we continuously invest effort on the most relevant and highest priority questions.
Question 3: Do you ever come across organizations where existing company culture does not allow experimentation to flourish?
Having worked with many corporate and enterprise organizations, we have found that in some cases, culture can be a major barrier to starting and scaling an experimentation program. In this case, an “intrapreneur” is challenging the status quo and well-established company norms. This situation often requires persistence to overcome.
Here are three key tips we have provided in the past to intrapreneurs trying to shift their organization to an experimentation mindset and culture.
As an internal champion/intrapreneur your should:
- Find a way to prove the value of experimentation by focusing on low hanging fruit that is not politically or technically constrained.
- Leverage outside perspectives to challenge existing company mindsets and current results
- Rigorously focus on determining the levers of change and implementation time that’s right for your organization
Question 4: For organizations that are at the beginning stages of implementing an experimentation-focused program. What are some key points/metrics that can be leveraged to increase the support/enthusiasm from others within the organization?
The exact approach to developing enthusiasm will depend on the specific situation, but here are a few considerations:
Those in the organization feel it’s a threat to the status quo and might subtly undermine efforts with anything new. Be aware of the fragility of the organizational support in the early days.
Conducting an “Allies and Detractors” exercise can help identify strategies for handling different groups. Then put a plan in place for how to utilize your Allies to support the cause.
It sometimes helps to let a new program develop with some air cover during its infancy to allow momentum to grow. The first few experiments might be kept quiet until there’s a solid, irrefutable success story to tell.
Once the core team’s momentum is strong, involving other adjacent groups to contribute ideas and insights will spread the enthusiasm. Ultimately, it takes time to shift the culture and process of decision-making throughout the organization, so follow a disciplined approach to ensure the little bird’s wings are strong and ready to take flight. The key is to understand the current state and the best next steps to involve the organization when the time is right.
Also, keep in mind, all of this gets much easier as the management in the organization already has the enthusiasm for this strategy. In situations where culture and buy-in need to be developed, we work with our clients to identify barriers using our PACET Assessment, which identifies specific recommendations to shift the momentum.
Question 5: Would you suggest an organization focus on tools or processes when starting an experimentation program?
Many companies prioritize technology over strategy, often hoping for a silver-bullet solution to deliver growth. The reality is usually disappointing. Knowing the right problem you’re trying to solve is critical for selecting and implementing technology properly.
That being said, technology is an essential component of running experiments. Your experimentation tool allows you to measure the statistically valid effects of any change you make and it has to integrate with the rest of your Martech stack and fit your organizational characteristics.
It is important to remember that tools used to run experiments are just one of the pillars that make up your Experimentation Operating System™ (EOS). There are four additional pillars to include—Process, Accountability, Culture, and Expertise.
To ensure you select the right tool for your business, we suggest considering how it fits within the context of your marketing ecosystem. Think about how you will use the tool and place equal emphasis on the accompanying process you will employ for effective execution and performance.
Question 6: How to help educate senior leadership to have a strategic approach to testing, to move past throwing spaghetti at the wall and seeing what sticks.
Often framing a narrative is an essential element when trying to get buy-in and build momentum. Understand what motivates them. Get to know their KPIs and be able to speak their language by talking about experiments, results, and insights in terms that matter to them.
Find low-hanging fruit and generate quick wins that can provide evidence to support the value of experimentation. We always recommend deferring politically sensitive topics to avoid potential roadblocks until you’re able to share meaningful insights.