Evangelizing experimentation: A strategy for scaling your organization’s test and learn culture

Evangelizing experimentation is just as important as execution in a “test and learn” culture.

If you only considered evangelizing experimentation as an afterthought to your testing program, stop. right. now.

I’m going to suggest something potentially radical—that you plan your communications strategy in tandem with your experimentation plan.

Building excitement for your program, creating buy-in for experimentation—these are necessary for proving your experimentation program’s value and for scaling your test and learn culture.

In a series of interviews with Lauren Schuman, Director of Growth at MailChimp, we saw firsthand the power of this approach as she moved from the initiating stage of experimenting, all the way through to scaling the program to an organizational initiative.

“A big part of successful experimentation is transparency and visibility. Visibility into what we’re doing and what we’re learning—and getting people really excited about that,” explains Lauren. “And that has contributed a lot to the success of where we are today.”

Evangelizing experimentation is crucial for getting executives and team at-large on board with a test and learn culture, so strategic communication is not only necessary…

It’s a priority.

In this post, learn how to evangelize experimentation to scale your organization’s test and learn culture.

I outline a four-part strategic communications framework for evangelizing experimentation at your organization—Inspiring, Informing, Involving, and Iterating.

I’ll also provide real-world tactics applied by Optimization Champions at digitally savvy brands like MailChimp, Student Brands, Wistia, and HubSpot. (So, we know they work.)

In the First Round Review article, “Master the Art of Influence—Persuasion as a Skill and a Habit”, Tyler Oldean, Product Leader for Google, likened persuasiveness to the dual process theoretical concepts of System 1 and System 2 thinking.

The idea is simple: We have two ways of thinking—the predominant fast, instinctual decision-making (System 1), and the slow, rational decision-making (System 2).

Tyler argued that we largely try to influence our colleagues and executive team with business cases focused on System 2 thinking; we gather data and evidence that takes time and understanding to process when we make a pitch.

And it’s true—we rarely make appeals to System 1 thinking in business. We tend to relegate the role of emotion to customers, not our teammates or our leaders.

But that’s how we inspire.

When it comes to evangelizing experimentation, you want to develop key messages about your program that incite enthusiasm. Storytelling, learnings, and insights can be a powerful motivator for your team to adopt the experimentation mindset.

That’s because communication (what we’re really talking about when we’re talking about evangelizing experimentation) is all about the audience.

It’s important to move beyond just presenting the evidence; you want to dig into your insights to create awe by considering:

1. The audience: Who are you speaking to?
2. The key messages: What do they need to hear?
3. The tactics and channels: How do they need to hear it for them to feel inspired?

But that takes intention.

We started from the very beginning and documented every single step of the workflow. We did a RACI model associated with that so that we were very clear on who was doing what; for example, who do we need to consult versus inform in the organization? And mapping the actual communication strategy, all the way down to ‘Is this a notification via Slack? Is it an email? Is it part of our monthly newsletter that we’ve created?’

– Lauren Schuman

As Lauren said, she uses the RACI model to determine key audiences for evangelizing experimentation. The model determines:

• Who is responsible?
• Who is accountable?
• Who should be consulted?
• And who should be informed?

The RACI model allows you to tailor your message appropriately for the audience and determine which channels make sense.

Planning out your communications strategy ensures that you stay on message and that you make the right appeal (based on System 1 or System 2 decision-making).

It also ensures that you don’t lose touch with those people on the periphery of experimentation—those who don’t understand the terminology; those who may be intimidated by your optimization team’s expertise; those who aren’t actively involved but could benefit from adopting the experimentation mindset.

Of course, communication is not a one-size-fits-all approach. You will need to tailor your approach to different teams/audiences in your organization.

Frame your experimentation as learning.

In my interview with Johnny Russo, AVP of E-commerce and Digital Marketing at Mark’s Canada, he expressed how people sometimes shy away from experimentation because of its association with success and failure.

Even though we are seeing more and more people embrace the concept, failure can still be perceived as a risk. For the risk-averse, it’s a frightening concept.

That’s why many Optimization Champions view experimentation, whether winning or losing experiments, as learning. They know that experimentation drives a deeper and deeper understanding of the customer through insights.

Andrew Capland, Director of Growth at Wistia, described how he framed an experiment led to new understanding, fueling excitement for the program:

“I said, going into this experiment, all my experience told me that the winner should be this variation. But I wasn’t really sure, because this audience is different,” explained Andrew.

“So here’s what I tested and here’s what happened. I can’t believe it, but I’m kind of humbled—I was completely off on this thing. This decision that I made, that I pushed in this meeting, actually decreased the business metrics or whatever it is that we were measuring by five or 10 per cent.

“And I just said it—Hey, I was wrong in this instance, but I know it. Imagine all the other decisions that we make, where we could be wrong and we have no idea. If you’re interested in doing things like this, reach out to me because we are running lots of tests.”

There are numerous ways to make an emotional appeal, to influence others with an inspiring message about experimentation.

 

Your customers are the hero. Tell their hero story.

Ultimately, it’s not about being right or wrong. It’s about helping our users see value in whatever way is on their terms.

– Andrew Capland

Like with any business change, when you evangelize experimentation, you are managing uncertainty within your team at-large. Your team may be hesitant about the change, in denial about the need to change, or resistant to change.

Because who has the time to figure out what this “experimentation” really means? Particularly, if it’s being communicated to you in a way that you don’t understand.

According to Lauren Schuman, evangelizing experimentation is dependent on not just your team’s technical expertise, but also on their social skills.

The team is really important, and mine is doing a phenomenal job. I think a lot of it is about the ability to evangelize experimentation across the organization. To do it in a way that isn’t in your face, and is very smart. It’s emotionally intelligent. In my hires, that was something that was extremely important—it wasn’t just about the technical capabilities and the skills. It was very much about finding the package of skills and the emotional intelligence.

– Lauren Schuman

Persuasion is an emotional appeal. And it takes emotional intelligence to be persuasive, to spread enthusiasm for your experimentation program. Lauren leverages her entire team to evangelize experimentation, spreading the idea, making it become more familiar.

At WiderFunnel, we know that the most effective way to beat the me-versus-them attitude, the silo mentality, is to take the focus off of your colleagues and yourself.

Because experimentation is and has always been about the customer.

Traditionally, business leaders attempted to guess what customers wanted. Informed by market conditions, their sales data, customer research, and their years of experience, they implemented new strategies with varying degrees of success.

If they failed to meet the customers’ expectations, conversion rates would decline, and leaders would have to guess, again, what went wrong.

But that bad decision was already implemented in their customer experience and they might make another wrong decision to correct the first mistake, only to lose touch with what the customer wanted.

As Ralph Chochlac, Director of Product at Student Brands, stated, experimentation makes choosing what the customer wants easier:

Decision-making becomes extremely easy once your experiments start revealing real user data. You’re no longer sitting in the boardroom making decisions based on gut feeling. The conversation shifts to data and hypotheses, and any idea that comes up is suddenly a candidate for an experiment. There is no HiPPO syndrome, because everyone realizes that it’s actually easier to do your job if you are running experiments and letting the data guide the way.

– Ralph Chochlac

Even with the focus on making data-driven decisions at Student Brands, Ralph is really highlighting a key message: experimentation makes the executive’s job easier.

He is not saying that they are wrong in conducting business based on gut feeling, but he is saying that experimentation, the introduction of this new method of doing business, makes their life easier.

Testing puts the user at the heart of every new strategy or initiative. Customers, whether they are faced with Variation A or Variation B, inform you of their preference by providing business leaders with comparative data of how the change affects the customer experience.

When you share your latest experiment, frame your message to show how a test and learn culture is bettering the customer experience. Position your results in terms of your customer’s satisfaction through your KPIs.

Customer centricity takes the focus off your team’s strengths and weaknesses, by showing what you can do for your customers.

 

At WiderFunnel, we often preach that as you’re just getting started with testing, one of the most powerful ways to evangelize experimentation is to share those quick wins. That’s because those types of results, especially as you are initiating your program, can create awe.

In the business, we call these experiments the low-hanging fruit—those simple tests with huge impacts. People take notice.

And they can be very powerful if you share the results internally.

Choose the communications channels that reach your team at-large:

To truly evangelize your experimentation program, you need to communicate the right message at the right time in the right channel. Adopt the internal communications channels that your team uses most.

On your intranet or internal wiki

Posting experiment stories, learnings, and insights from experimentation program can lead to deeper engagement with your program. And if your intranet supports multimedia, you can be creative with a podcast, a video, or even adding images and graphics.

We have a wiki for internal communications. We’re really trying to get away from “success theater,” So every time I run any experiment, whether it wins or fails, I will do a little recap or a post-mortem. I will say what happened in a narrative—what we were thinking and maybe where we can go next.

– Alex Birkett, Growth Marketing Manager, HubSpot

As Alex says, move beyond the “success theatre” and focus on what makes a compelling story for your audience. Put your customers at the center and create awe by highlighting experiments with unexpected results or interesting findings or learnings.

If you don’t have an intranet, you can achieve the same result your team collaboration software – within a Slack channel, for example.

Through an internal newsletter

Broadcasting your latest news, results, and resources can help to champion your experimentation program internally. Lauren uses an experimentation Slack channel to drive internal momentum, using the communications channel that the team at-large is accustomed to using.

We also created an internal newsletter using MailChimp. Employees can subscribe to the monthly newsletter to stay informed on what experiments ran and what the learnings are. We also share articles that we’re reading in the growth and optimization space to keep people engaged.

– Lauren Schuman

At your meetings

Whether you meet weekly, monthly, or quarterly, anytime you get a group of people together in a room is a great time to share insights from your latest experiment.

You can provide a presentation, share insights and learnings, or even walk-through an experiment from ideation to the results analysis like Ralph Chochlac does at Student Brands.

We do something called demos which is an opportunity for everyone on the team to show what they’re working on, the tools they’re using, the technical details involved, and what they learned as a result. For example, we’ll dive deep into recent experiments, how they were constructed, coding challenges, and edge cases.

– Ralph Chochlac

Face-to-face conversations

Even if informal, don’t neglect the power of the face-to-face conversation. Andrew Capland from Wistia will bring key people from other departments into an informal meeting room with a whiteboard, to start brainstorming ideas and explaining the experimentation protocol and process. These ad hoc sessions can inspire others.

Through an internal dashboard

If your team is largely data-driven, speak their language. Many optimization teams build their own internal dashboard to show the KPIs for their program, so your executive team can keep a pulse on what’s happening with your experimentation program.

Through an employee bulletin

If you have a bulletin board in your break room, give your team some interesting reading material. Print off your latest intranet story or share some of your latest results when people have the time to read it. We’ve even heard of a company that posts experiment results in the washroom to catch people’s attention.

—

Evangelizing experimentation doesn’t have to be overly creative when it comes to the communication channels and media that you use to share your message for a test and learn culture.

But it has to work.

If you have moved past the low-hanging fruit, your ability to evangelize experimentation may require more engagement from your team at-large.

Evangelizing experimentation is about building a desire for your team at-large to participate in your program. And we enable this change in our team by building up their core copetencies as you scale your experimentation program to a business-wide strategy.

As you develop a test and learn culture, you will want to ensure that your experimentation protocol is documented and accessible. But there are other ways to get your team involved, to get your team invested in your vision for a test and learn culture.

Whatever your tactic, make it fun, communicate in a way that allows them to engage more deeply in your experimentation program.

 

Feed curiosity to create a test and learn culture.

The foundation of a test and learn culture is curiosity. It’s one of our core values at WiderFunnel, because we have a test and learn culture that fuels our experimentation with North American brands.

According to Andrew Capland, by documenting experiments through the lens of learning, you influence people’s perceptions about experimentation:

“I don’t think about an A/B test as a failure. I just think about it as a learning. I almost use it as a proxy when I talk about it. I designed an experiment so that we physically type up: If this experiment is successful, what do we learn? If this experiment is not successful, what did we learn?” Andrew described.

“So, that no matter what we’re constantly learning and getting better over time. And I think it’s absolutely created a culture of learning from this where I’ve seen in other areas of the organization where our product team will work on something and they’ll say, you know, we were really excited about this, but it just didn’t work out; Here’s what we learned.”

A powerful way to encourage people to adopt a test and learn mindset is to provide more opportunities for people to learn. To learn about experimentation protocol, design of experiments, analytics—the list goes on.

Once they are motivated to experiment, you’ll want to ensure they know how to change, that they have the skills and abilities to contribute to your program in the capacity that you see fit.

Whether that’s simply adding to your idea backlog or actually running experiments within their team, education is necessary in getting your team involved in experimentation.

Make continuous learning a priority through education initiatives.

There are traditional and innovative ways to educate your team about experimentation. You could:

• Share resources, articles, and academic papers with the latest research and news about experimentation. (This works for MailChimp!)
• Host a Lunch & Learn with reputable internal and external experts on experimentation. (This works for Hubspot!)
• Review your user research as a team, to highlight the customer’s experience, and show how to formulate hypotheses based on your team’s findings. (This works for Wistia!)
• Plan a workshop, summit, or training session with your team to encourage learning and development. (This works for Mark’s!)
• Schedule a Show & Tell or a Demo, to walk through an experiment and highlight the insights that were gained whether the test won or lost. (This works for Student Brands!)
• Try something new: optional hackathons, solution brainstorms, book clubs, and even ideation sessions inspire your team to get involved. (This works for WiderFunnel!)

Education can fuel your team, leading to better ideation, innovation, and iterations of your experiments.

Leverage 2-way communications to make engaging in your experimentation program a game.

Gamification can be a highly motivating way for your audiences to become involved with experimentation through two-way communications. What this means is that you bring the elements of a game—scoring, reward, and competition—into your internal communications.

For Lauren Schuman, gamification is as simple as offering a poll on MailChimp’s internal Slack experimentation channel:

“My favorite method [for evangelizing experimentation] is our Slack channel, #optimization. Every time we meet somebody, or we’re in a cross-team meeting, we drive people to join the channel and the conversation. It’s a very active channel with nearly 100 people in it.

“ Every time we launch an experiment we try to have some fun with it. We post the experiment in our Slack channel and we ask people to vote on what they think the winner will be. We’ll give some details, like we’re launching experiment XYZ, what do you think will win – A. The Control, B. Variation 1 or C. Variation 2? And people can actually vote via the poll feature in Slack.

“Then, when the experiment wraps up, we post the results, and, of the people who guessed correctly, we pull a name out of a random generator and give them a Chimp coin. The Chimp coin unlocks the Rainbow Room where the winner can pick out some MailChimp swag. It’s a fun way to engage people, get them to care about the experiment, to pay attention, to understand the results, and celebrate the wins.”

Voting on winning variations is something we do at WiderFunnel at our monthly A-ha! sessions, where we celebrate the client experiments that had the most impact or insights.

But even if you can’t offer a reward in terms of swag like at MailChimp, there’s value to recognize the major players in your experimentation program, even those who contributed ideas from other teams.

“Every Friday, my team writes down our learnings for the week on our internal wiki,” explains Alex Birkett.

“We can tag people if they’re related to the story and we have a bunch of followers who are reading along with us. People can comment, or ask specific questions. If there’s a big test – if it’s something that takes a lot of resources and there’s a lot riding on it, and especially if it wins—we might write more like a blog post on the wiki. We’re trying to lead by example.”

Whatever your tactic, make it fun, communicate in a way that allows them to engage more deeply in your experimentation program.

Communications—like experimentation—is an iterative process. You’ll need to revisit each part of the communications framework (inspiring, informing, involving, and iterating) as your experimentation program matures. That’s why it’s good to evaluate your strategy to evangelize experimentation to see if your messages and channels are sticky.

You can measure your strategy through:

1. The number and quality of idea submissions to your backlog;
2. Engagement and attendance at your education workshops, summits or Lunch and Learns;
3. The volume and sentiment of comments or questions you receive about experimentation;
4. Even, the participation in your online voting polls.

And as you scale your test and learn culture, you’ll want to ensure that you can integrate new team members quickly, in a way that maintains their enthusiasm.

“You have to worry about organizational indicators of success like knowledge sharing, especially when you think about how many people they’re hiring and how many people are changing teams. You have to worry about how to get them up to speed faster so they’re not reinventing the wheel and not re-doing stuff that’s already been done,” explains Alex Birkett.

“How unmotivating is that to just constantly run into that wall where you have no idea what’s been done and what you should do?”

For one, when someone engages in your experimentation program by asking a question or submitting an idea to your backlog, it’s important to be mindful of their enthusiasm.

Even if their ideas are not hypothesis-ready, even if their ideas are not backed by enough evidence—make sure that you are communicating in a way that inspires, informs, and involves them in your experimentation program.

Because that’s how Optimization Champions are formed.

How have you evangelized experimentation at your organization? We’d love to hear about your insights in the comment section below.