Moving the needle: Strategic metric setting for your experimentation program

Is your experimentation program actually moving the needle when it comes to growth?

At the heart of any strategy is evaluation and measurement: Are you getting to where you want to go? Are you actually moving the needle when it comes to growth?

In the “State of Experimentation Maturity 2018” original research report, we discovered that less than 50% of all respondents had metrics set for their experimentation program from the executive team.

There are real consequences to not having your executive team involved in your experimentation program’s metrics:

• Your optimization team may end up working in a bubble
• You might be solving problems that aren’t a priority to your executive team, seriously sidestepping what your business should be caring about
• “Testing” may be seen as a side initiative within your organization, instead of the business-wide strategy that experimentation warrants

But to scale your experimentation program, to gain internal buy-in from your team at-large, you’ll need to ask yourself: What metrics do my executive team care about?

Strategic metric setting provides a system of accountability; it ensures your team has eyes on the right goal, at the right time. And that you are accelerating your business goals and driving growth for your organization.

It makes certain leaders are paying attention and will continue to pay attention to experimentation.

And senior-level buy-in can lead to increased knowledge-sharing and the application of your insights across the business. It can also result in increased resourcing for your team, including more development, design, strategy, or data analyst support.

You can take those first real results back to your executive team and say, “Look what we were able to accomplish in terms of your business priorities!”

And that’s what moving the needle is all about.

In this post, you will grasp:

• Why it is essential to align experimentation metrics to your organization’s business goals;
• Effective metric-setting frameworks that allow you to strategically waterfall your KPIs from those top-level goals.

I will also provide insights from Optimization Champions at brands such as Tsheets by Quickbooks, Wistia, Pagely, Salesforce, and Student Brands on how to align their experimentation metrics to achieve true organizational growth.

While a company is growing fast, there is nothing more important than constant communication and complete alignment.

– Marc Benioff, CEO and Chairman, Salesforce

Mark Benioff, CEO of Salesforce, created unique management process called V2MOM to align his teams toward business growth. He credits the V2MOM tool as what helped Salesforce remain agile in a digital world.

A V2MOM is an acronym comprised of five components:

• The Vision for what you want to do or achieve as an organization;
• Values that help you pursue the vision;
• Methods for how you will get your vision accomplished;
• Obstacles that you need to overcome to see your vision realized;
• And Measurement, the results of achieving your vision.

How the process works: the executive team fills out their document, and that document is passed down to the leaders of internal teams.

Those leaders design the methods and measures that cascade directly from those organizational goals in their own V2MOM, so that the entire team is focused on achieving that ultimate vision.

But they also share their V2MOM document laterally (think peer-to-peer or manager-to-manager), ensuring internal alignment.

As such, it can be used for any phase in the business—from quarterly goal-setting to communicating annual priorities.

When we wrote the post, “How to blast through silo mentality and create a culture of experimentation”, we talked about aligning your internal teams on a vision for experimentation. A V2MOM is exactly the type of tool that is needed to bring everyone on board. It enables every team to work in complement to each other through their goals and metrics, rather than pitting internal siloes against each other with competing priorities and goals.

When it comes to strategically setting metrics for your experimentation program, think of how the V2MOM provides strategic direction at Salesforce. You can adapt and evolve your own method for aligning your organization at-large, but to really demonstrate how experimentation can drive growth, look to your metric-setting strategy.

While the V2MOM provides a simple framework for decision-making and internal alignment, Optimization Champions at various organizations employ similar methods for goal-setting:

We set goals and metrics with our executive team on a quarterly basis, which are then translated by the team into specific initiatives and experiments.

– Ralph Chochlac, Director of Product at Student Brands.

“Sometimes experiments uncover opportunities that we had not thought of before, and so this gives us the chance to double-down and explore further,” Ralph explains. “These insights also feed back into future goals and metrics. For example, the more we understand user behavior, the more refined our metrics become over time.”

Just because you have a strategic plan, as both Marc Benioff and Ralph Chochlac indicate, doesn’t mean you are restricted to that plan indefinitely.

Today’s digital world requires that your business be agile.

But the most important factor in mitigating rapid change is ensuring the organization at-large is aligned on what the business needs to achieve.

What makes measurement so potent is its capacity to instigate informed action—to provide the opportunity for people to engage in the right behavior at the right time.

– Dean R. Spitzer, author of “Transforming Performance Measurement: Rethinking the Way We Drive and Measure Organizational Success“

Before you waterfall your experimentation metrics from your business goals, let’s get clear on the difference between business goals, metrics, and your key performance indicators.

Your Business Goals

Your business goals are pivotal to the business plan or strategy. That’s because your business goals are where your organization wants to be. For instance, your business goal may be to improve the customer experience, increase brand loyalty, and drive revenue and growth.

Business goals are not measurable in themselves. Your tactics (in your case, experimentation) must align with these goals.

Your Metric(s)

Your metric(s) break down business goals into measurable outcomes for each team. They are tools to gauge whether your strategic plan—including the goals and the methods you have outlined to reach those goals—are working in terms of both process and progress.

Your metric can have more than one measure to it. For instance, your metric for the above goal may be to focus on the customer’s lifetime value, but there are many ways to gauge this metric and that’s where your key performance indicators come in.

Your Key Performance Indicators

Your key performance indicators (KPIs) gauge your progress or growth. They let you keep a pulse on your strategy so that you can see if you are on track or off track.

You may evaluate a customer’s lifetime value through certain KPIs: the average customer lifespan, the customer retention rate, or average gross margin per customer’s lifespan.

But you might also be looking at the frequency of visits or purchases, the engagement rate in your loyalty program, and even the cost of customer acquisition and retention.

Your business is unique: a valuable customer experience is dependent on different factors in your business context.

Some optimizers call your metric the True North Metric or the North Star Metric. But what they are really referring to are the metrics that have the most value to you and your customers.

“Every business group must define a suitable (usually composite) evaluation metric for experiments that align with its strategic goals. That might sound simple, but determining which short-term metrics are the best predictors of long-term outcomes is difficult. Many companies get it wrong,” explains Ron Kohavi and Stefan Thomke, in the Harvard Business Review article, “The Surprising Power of Online Controlled Experiments”.

“Getting it right—coming up with an overall evaluation criterion (OEC)—takes thoughtful consideration and often extensive internal debate. It requires close cooperation between senior executives who understand the strategy and data analysts who understand metrics and trade-offs.

“And it’s not a one time exercise: We recommend that the OEC be adjusted annually.”

When you start to dive into what it means to reach your business goals, you will have to spend time debating and considering different metrics and KPIs.

You need to understand the risks and rewards for each of your proposed metrics.

When you are trying to gauge the validity of your metric, try to consider multiple viewpoints. Invite your executive team, your data analysts, and your optimization strategists into the conversation.

Having multiple viewpoints on the potential pitfalls and opportunities only helps clarify that you are focused on the right metric. (Think cross-functional teams!)

When it comes to metric setting for your experimentation program, the first question you can ask yourself is if your metric reflects your value proposition.

There’s a reason we put the value proposition at the center of the LIFT Model®. Your value proposition is what incites your customers to act. And creating a more delightful customer experience helps you move the needle.

In his post “The Science of Quality Growth”, Ken Soong, Data Scientist at LinkedIn, described how they created a custom metric, called “Quality Signup.” The experimentation metric gauges the users who have made concerted efforts to create an online identity and make connections in the six months following registration.

“The first step in creating a custom true-north metric is to understand how your product aligns with the value propositions your business provides. Since metrics serve to measure and quantify a product against an existing goal, they are ineffective without strong alignment to the value propositions,” articulates Ken Soong.

Customer acquisition limits their understanding of how customers are engaging with the platform. LinkedIn could not gauge whether the customer was getting any value out of the experience, their product.

Instead, the Quality Signup metric points to the customer experience, rather than solely conversions.

Such custom metrics are not unique. A True North or North Star Metric ensures that you are not relying on vanity metrics, but ensuring you are optimizing for the entire customer experience.

“Every customer who shops at your business has some lifetime value. They may shop there once, or they may shop there every month.“- Mike St. Laurent, Optimization Strategist and Product Development Lead at WiderFunnel.

“This distinction is important because while you may be looking at session-based conversion optimization, the best companies in the world are optimizing for customer lifetime value and profitability.”

When it comes to a customer’s lifetime value, notable brands have different custom metrics. For example:

• Facebook optimizes for new users who make 10 friends in seven days.
• Airbnb focuses on the number of nights booked by guest.
• Slack tracks teams who send 2,000+ messages.
• And Pinterest hones in on Weekly Active Repinners.

If the users are actively pinning, messaging, booking, and connecting with others, these metrics can provide a solid indication that they are finding value in the product.

But each business is unique and it’s important to understand what the ideal customer experience looks like for your business context. And you may need to refine your approach as you iterate your way to a deeper understanding of who your customer is and what they prefer in their experience:

“We used to focus on typical conversion metrics such as registration and subscription rates. But being a content business with a significant SEO component, we realized we had to optimize for the entire user experience,” explains Ralph Chochlac.

“This meant developing new metrics that truly reflected how users were engaging with our content, whether they found it valuable, whether they were coming back, and so forth.

“As a result, we evolved our definition of experiment win-rate to balance all of these aspects, and not simply monetization.”

For Ralph Chochlac, re-evaluating the experimentation metrics made certain that Student Brands was providing the best value to their customers.

While registration and subscription rates are conversions, the metrics didn’t relate to the customer experience—whether they engaged with the content, returned to the site, or even developed an affinity for the brand.

In the article, “The Overall Evaluation Criterion (OEC)”, the author Ron Kohavi advised that “key metrics should be hard to game”. He advises against focusing only on one key metric to the detriment of your business.

Over the course of his research, he found several examples of how metrics have led to negative business impacts.

He recalled how one restaurant manager aimed to get a perfect 100% score an internal efficiency ratio for food ordering. The manager wanted to ensure that every menu item was cook to order to avoid food waste.

While the manager got the perfect score, the customer experience suffered; long wait times eventually caused the business to shutter its doors.

When people are focused on achieving a single metric—particularly if it is the wrong metric or if the metric can be “gamed”—they are at risk of tunnel vision.

That’s why keeping a pulse on their specific metric while also understanding other factors at play is central to the analysis that takes place at Tsheets.

“We want to get more detail around the conversion metric. Having someone that owns and is accountable to the conversion metric is a highly visible place to be with opportunities, but it’s also a two-edged sword,” explains Mike Loveridge of Tsheets by Quickbooks.

“It’s kind of dangerous if you don’t have the details of why the conversion rate dips and what is really affecting it: if it’s a whole page or part of a page; if it’s external factors or market factors; if it’s traffic related or PPC—you’ve got to really be able to drill down.”

Ron Kohavi suggests focusing on the customer lifetime value, when you are trying to set metrics for your experimentation program. This means that you are moving beyond vanity metrics and working to actually improve the customer experience.

“Anytime a metric can be trending positive without a corresponding improvement in overall value delivered to customers, it is probably not a good North Star Metric,” Sean Ellis reasoned in “Does a North Star Metric Oversimplify Growth?”

So don’t get distracted by metrics that don’t have an impact on your business.

With the digital transformation of your business, you accumulate an immense amount of information. So, these vanity metrics are easy to get lost in.

Having clear metrics provides a directional compass for your data analysis.

There’s such a huge amount of data that when you don’t have some sort of guiding force that works as an anchor, it becomes absolute chaos. There are a lot of proxy metrics that only sound like improvements, but don’t improve the ‘North Star.’

– Chad Sanderson, in “Website conversions and the ‘North Star’ metric: Q&A with Subway’s Chad Sanderson“

As you dig deeper into your experimentation metrics, you may refine your strategy for how you measure success and for what actually has value to your organization.

And it’s important to remain agile.

Be focused but open your mind new ideas and insights into how you can drive your organization’s top-level priorities.

In Ron Kohavi and Stefan Thomke’s HBR’s article, “The Surprising Power of Online Controlled Experiments”, the authors demonstrate the power of prioritization with a compelling example.

While working on Microsoft’s Bing search engine, an employee evaluated the effort required to create an experiment based on one of their backlogged ideas. And the results were astounding: a 12% lift—all deriving from a few lines of code that the experiment required to run.

The story highlights the importance of a multi-pronged prioritization framework.

For WiderFunnel, we created the PIE framework, adopted by countless organizations around the world. The PIE framework evaluates experiment hypotheses based on three criteria: potential, importance, and ease:

• Potential: How much change can you have?
• Importance: How much impact can this change effect?
• Ease: How easy can your team make this change?

Reflecting back on Bing’s winning experiment, weighing these three criteria against eachother is the job for any Optimization Champion. While the potential and importance of the experiment was largely underestimated, the ease of implementation allowed the hypothesis to rise to the top.

And framework thinking can help you make solid, data-backed decisions when it comes to knowing what tests can make a true business impact.

A proper prioritization framework not only ensures that you are allocating the right resources to the most impactful experiments in terms of your goals. It also makes sure that your team is strategically focused on driving your organization’s growth.

When you have a well-defined metrics and solid prioritization process, you encourage data-backed decisions from all organizational levels.

Andrew Capland, Director of Growth at Wistia, recalled how a commitment to making data-backed decisions means arguing against the CEO’s experimentation idea when it matters.

“Our CEO Chris Savage one time said to me, ‘I view my job as sharing ideas with you and I view your job as evaluating my ideas and then making the right decision.’ And I thought that that was incredibly empowering,” explained Andrew Capland.

“And what I did is I basically gave him transparency into the decisions we were making by sharing our prioritization framework. I would sometimes shoot him a quick note: “Hey, you added in a few ideas in our Master List and we used our framework and graded it, and we’re not going to do it. We’re going to do this experiment because based on our goals, we think that’s the most impactful work. But keep the ideas flowing.”

“And he actually really liked it, to his credit, and I thought he did a really great job of saying, “Hey, thanks for letting me know. That’s exactly what I want you and the team to do. We want you to make the right decision and not just stop everything and do it because it’s my idea. You’re closest to it; you’re the ones responsible for the results.”

Once you have your metrics and KPIs set, you’ll want to devise a system for tracking and sharing your results. Centralizing this information so that everyone on your team has access it will only further solidify your business goals into their daily routines, into their workflows.

Rod Austin, Director of Marketing at Pagely, understands the impact of tracking your metrics in a centralized place. He explains:

“We recently made sharing the most important department results a more formal process company-wide. Every department head reports in a Slack add-on (called Standuply) their weekly KPI’s and a brief summary of what their team accomplished that week,” described Rod Austin.

“This is a great way to share what we’re working on in the same place as our day-to-day conversations. Before our weekly meeting, everyone can review the metrics from every department, marketing included, to get a quick summary of what teams have been up to, maximizing our time on the team call.”

He wasn’t the only Optimization Champion to think tracking your goals in a centralized place is beneficial. Mike Loveridge also described how having a set experimentation metric provided a way of aligning on the business goals each morning, and making educated guesses as to where they could be in the upcoming quarter.

“We are currently constructing a dashboard and putting in the background data in place to support owning that conversion metric. It’s always been a metric that we’ve looked at on a daily basis but in the near future, we’ll really be able to analyze and forecast off of that metric as well.”

Giving your executive team the ability to track your progress allows them to keep a pulse on your activities. This can only benefit your program. Every time they check in, they are coming to realize the power of experimentation in digitally transforming their business.

Because experimentation is not solely about what is happening now, it’s a way of evolving to the future.

What is your True North Metric? We’d love to hear why you chose this measure in the comments.