Are you a product builder or a maximizer?
Hila Qu, Vice President of Growth at Acorns, has a theory. She says there are two kinds of product managers: the builders and the maximizers. Ideally, on your product team, you have both: the builders who conceptualize a solution in product development, from the ideation stage all the way until the customer roll-out.
Product Builders are essential to organizations because they conceive the product that provides value to the customer in the first place.
“They’re very important because, without them, there’s nothing; they are building the value from zero to one,” Hila explains.
And then, there are those that can maximize the customer’s value through experimentation in product development.
The bulk of the product value is already created, but how can you maximize that value? The maximizers help customers understand the product better when a new product or feature is launched because we often assume people will know how to use it.
Hila Qu, VP of Growth at Acorns
Product Maximizers build upon the value of the product through experimentation.
It’s an interesting distinction—between developing and experimentation.
Product development is all about making your product solve your customer’s real-world problem, and the builders aim to make the process frictionless so they will create a habit out of returning.
But product development teams might not know if they were successful. Or how they could improve the experience for the users.
Experimentation enables the maximizers to improve upon the value that is already developed, to make the customer experience more delightful.
Actually, your product team can leverage experimentation where ever they are in the product development cycle: They can test an idea before launch; Or they can improve your feature after it’s rolled out to your customers.
Because experimentation is a feedback loop with your customers. It moves beyond conversions or customer acquisition, to improving adoption and retention so that your product indispensable in the customer’s lives.
Experimentation is iterative; your product team learns and develops as they gain insights about your business and your customers.
If you are exploring the possibilities of product experimentation at your organization; if you are a marketing leader looking to evangelize experimentation throughout the business—this post is for you.
In this post, we’re tackling the essentials of experimentation in product development. Essentials like:
- The breadth and depth of experimentation in product development so you can start dreaming up the possibilities.
- An understanding of the stakes when experimenting within the product development cycle, especially when those hard-earned customers are interacting with your brand.
- And why experimentation in product development is a leap toward a customer-centric organization, a strategy for growth.
But we also wanted to show you examples. So, we talked to product managers at Optimizely, Acorns, RVShare and Stitch Fix about the opportunities for product experimentation in reaching business goals.
And they are clearly seeing results.
You might be thinking: what is product experimentation?
Client-side experimentation is based on the premise that what-you-see-is-what-you-get. It’s about visual changes to the content hierarchy of the website.
Innovative marketers have been doing this form of optimization for years, and they’ve seen the success. So what’s next?
Product experimentation is often top of mind when it comes to expanding an optimization program internally. That’s because there are more possibilities.
“The product life cycle is usually more elongated than the marketing lifecycle,” explains Giannis Psaroudakis, Director ofProduct at Optimizely.
“In the marketing lifecycle, you have a campaign and you have a very specific goal, which is sometimes short-lived – you want to increase your leads, for instance.”
If you are optimizing your website to get a customer to convert, only to have a product that does not live up to the customer’s expectations—those hard-to-win customers are going to look elsewhere.
But it can be tricky.
Product managers don’t always get it right the first time, and that’s why it’s important to have a feedback loop with your customers.
Product development teams need to work in an agile manner, testing and learning so that they can continuously evolve their ideas to meet the needs of their customers.
It’s important to maximize the value of your product. Understanding how your customers are finding value and optimizing their journey so that they can find that value faster.
Even well-researched products can suffer due to the gap between what customers think they want and what their behaviors reveal they actually want.
Optimizely: Getting Started with Product Experimentation
By testing a new feature or even a variation of that feature, your product team can see if it is improving the customer experience, or actually making it worse.
The experimentation mindset aligned with the premise of a minimum viable product, seeing how your customers react through a gradual or even incremental roll-out. It allows you to test new ideas before a full launch.
Don’t frame it as a product launch. Just frame it as an experiment.
Dan Siroker, Co-Founder and Executive Chairman, Optimizely
If you crave that buzz of a big launch, experimentation after-the-fact enables you to get the most out of those ideas—and your organization’s investment.
By the time something this big has been built, the launch is very, very unlikely to be permanently rolled back no matter what the metrics say.
Carson Forter, Data Scientist at Twitch in his Toward Data Science article: The Three Types of A/B Tests
“Rather, the randomized experiment, in this case, is for visibility, and to provide information that might help with making future decisions.”
Announcing a feature or product through an official launch can inspire adoption—particularly if you have a large customer base to which you can communicate.
But once your product is rolled out, you can start to understand how your customers are interacting with the features at every touchpoint through experimentation.
Product experimentation should make the job of the end user easier or more helpful. If that’s not the goal in your company than you might be running it the wrong way.
Martijn Scheijbeler, Vice President of Marketing, RVShare
Those learnings can be brought forth into future product evolutions—whether that’s changing copy to highlight your value proposition, or smoothing out the funnel flow to reduce friction.
And you can even experiment after your customer logs into your experience, especially when confidentiality is essential for your business.
The scientific method of product experimentation
Let me first clarify: you don’t test features in product experimentation, you test hypotheses.
Product experimentation takes an optimizer’s brain: It takes an analytical mind to be able to see the opportunities for product experimentation. It also needs the scientific method to turn ideas into measurable hypotheses.
Move away from the mental model of thinking of new products and new features you want to build, in the form of a list of requirements, and instead, think of them as hypotheses.
Giannis Psaroudakis, Director of Product, Optimizely
“Rather than saying, ‘we need to build feature X that has this requirement and that requirement, because customers asked us to’, you start with presenting these same requirements in the form of if/then statements.”
This is so important because it sidesteps any HiPPO (highest paid person’s opinion), and it persuades you to think about your product—how you’re building the product—in the form of experiments. In other words, hypotheses that you can validate with data.
Making data-driven decisions is important in product experimentation because the stakes are higher.
You are experimenting within your product experience on your many already acquired customers so you need to have high quality assurance and more guard rails to what makes it to the experimentation stage.
“We have a more rigorous review process for product experiment hypotheses,” recalls Giannis Psaroudakis.
“We have an experiment review, which is more of an advisory process that allows us to surface new product experimentation ideas and specifically review our hypotheses.”
“Anyone in the organization can submit their “experiment briefs”—customer challenge, hypotheses, and metrics of success—through Optimizely’s Program Management platform.”
“And we revise these experiment briefs in the weekly experiment reviews to make sure that whoever is planning to run a product experiment has carefully thought about the hypothesis and the metrics of success.”
A solid scientific method focused on solid hypotheses and design of experiments ensures that you can track how your customers are adopting your new features or product.
The scientific method ensures that you can make the data-driven decisions on how your product—and the customer experience—should evolve.
Experimenting broadly and deeply in product development
When it comes to experimentation in product development, you have more opportunity to optimize your experience. That’s because you can experiment broadly and deeply into the product development cycle.
There are plenty of opportunities to experiment in the product development cycle.
First, you have the Discovery stage where you ideate new features or products and this is particularly relevant when you ideate in the form of hypotheses.
According to Meg Watson, Product Manager at Stitch Fix, her team also conducts an analysis to determine if they should proceed into product development.
Next, you move into the Design stage, according to Giannis Psaroudakis, where you can evaluate your hypotheses through early user studies or rapid prototyping. This prototyping is where experimentation begins in terms of usability.
Then, you go on to the Development stage where you dive into coding or testing your actual code.
And afterward, you have the Roll-out stage, when you make sure your hypotheses still stand after launch and there’s nothing negatively affecting the quality of the final product or the customer experience.
In traditional product development, this is where you might stop. Your ideas might win, but they also might fail. And there’s no defined evaluation for where you can improve.
That’s why in product experimentation, you also have the Maximizing stage, that Hila Qu described: What can be improved? What can we evolve? What can make the customer experience frictionless, more delightful?
“When I’ve looked at some of the things we’ve started with and usability testing, and then compare that to the product that ships, X months later, a lot happens in between those stages. We start with sending a prototype through usability and then we have all the different optimization tests that run after something goes live.” confirms Meg Watson.
Throughout the product development cycle, with the experimentation mindset, you are constantly exploring and validating the possibilities for new features or products by generating that feedback loop with your customers.
But perhaps more importantly, experimentation in product development means you can test deep in your stack.
Unlike client-side experimentation, with its what-you-see-is-what-you-get approach, you can test what you can’t see. Product experimentation incorporates server-side testing:
“You can test changes on your server side and improve the performance of your back-end systems,” Giannis Psaroudakis affirms.
If you’re running any machine learning models that give recommendations to customers, you can experiment with those, for instance.”
But in server-side experimentation, the variation is coded on servers. So, it can be more resource intensive because you build up-front.
You can see that server-side experimentation requires that you build upfront. (Source: “Why Experiment Server-Side?” by Optimizely)
That way, you can experiment with how a product functions, like the machine learning algorithms that enables you to deliver a personalized experience.
And that’s why product experimentation has so much potential.
Learn more from these innovative organizations
- Stitch Fix’s tech blog: Multithreaded
- Hila Qu’s Medium articles on growth
- Optimizely’s blog on product development
When it comes to experimentation, it’s all about the customer experience.
Ultimately the goal of all this, the reason we’re doing this, is to give the customer a better experience with Stitch Fix and to make sure that it’s effortless. It’s fun. it’s delightful. And that they truly have a good experience with all of our products.
Experimentation in product development allows you to get deeper into the stack, deeper into your customer’s experience so that you are delivering the best possible solution and a delightful one at that.
For many organizations, it’s an untapped source of growth.
The gap between product and marketing teams is becoming smaller and smaller, and from a customer experience perspective, it’s blended.
“The customer experience has to be consistent and compelling, and work seamlessly between the pieces that are controlled by the two teams, in order to have the customer finish that journey from a random visit or from your ads, to the point where they use the product for the first time and they know the value, they see the benefit.”
If the customer doesn’t view marketing and product as separate parts of their journey, organizations shouldn’t either.
But we need to stop siloing product and marketing.
With an experimentation mindset, organizations need to unite internally to spread those insights throughout every customer touchpoint.
That means experimenting constantly—both client-side and server-side—and sharing the insights across both marketing and product development teams.
Because your customers don’t see your landing page or your product funnel as distinct experiences. Instead, you need to focus on the entire customer journey.
Subscribe to get experimentation insights straight to your inbox.