Experimentation in product development: How you can maximize the customer experience

10 min. read  | Last updated: September 23rd, 2018
Reading Time: 10 minutes

Are you a product builder or a maximizer?

Hila Qu, Vice President of Growth at Acorns, has a theory.

She says there are two kinds of product managers: the builders and the maximizers.

Ideally, on your product team, you have both: the builders who conceptualize a solution in product development, from the ideation stage all the way until the customer roll-out.

WiderFunnel Product Teams as Builders
Product Builders are essential to organizations because they conceive the product that provides value to the customer in the first place.

They’re very important because, without them, there’s nothing; they are building the value from zero to one,” Hila explains.

And then, there are those that can maximize the customer’s value through experimentation in product development.

The bulk of the product value is already created, but how can you maximize that value? The maximizers help customers understand the product better when a new product or feature is launched because we often assume people will know how to use it.

Hila Qu
VP of Growth at Acorns
WiderFunnel Product Teams Maximing the Value of Customer Experience
Product Maximizers build upon the value of the product through experimentation.

It’s an interesting distinction—between developing and experimentation.

Product development is all about making your product solve your customer’s real-world problem, and the builders aim to make the process frictionless so they will create a habit out of returning.

But product development teams might not know if they were successful. Or how they could improve the experience for the users.

Experimentation enables the maximizers to improve upon the value that is already developed, to make the customer experience more delightful.

Actually, your product team can leverage experimentation where ever they are in the product development cycle: They can test an idea before launch; Or they can improve your feature after it’s rolled out to your customers.


Experimentation is iterative; your product team learns and develops as they gain insights about your business and your customers.

If you are exploring the possibilities of product experimentation at your organization; if you are a marketing leader looking to evangelize experimentation throughout the business—this post is for you.

In this post, we’re tackling the essentials of experimentation in product development. Essentials like:

  • The breadth and depth of experimentation in product development so you can start dreaming up the possibilities.
  • An understanding of the stakes when experimenting within the product development cycle, especially when those hard-earned customers are interacting with your brand.
  • And why experimentation in product development is a leap toward a customer-centric organization, a strategy for growth.

But we also wanted to show you examples. So, we talked to product managers at Optimizely, Acorns.com, RV.com and Stitch Fix about the opportunities for product experimentation in reaching business goals.

And they are clearly seeing results.

You might be thinking: what is product experimentation?

Client-side experimentation is based on the premise that what-you-see-is-what-you-get. It’s about visual changes to the content hierarchy of the website.

Innovative marketers have been doing this form of optimization for years, and they’ve seen the success. So what’s next?

Product experimentation is often top of mind when it comes to expanding an optimization program internally. That’s because there are more possibilities.

The product life cycle is usually more elongated than the marketing lifecycle,” explains Giannis Psaroudakis, Director, Product at Optimizely.

In the marketing lifecycle, you have a campaign and you have a very specific goal, which is sometimes short-lived – you want to increase your leads, for instance.

If you are optimizing your website to get a customer to convert, only to have a product that does not live up to the customer’s expectations—those hard-to-win customers are going to look elsewhere.

But it can be tricky.

Product managers don’t always get it right the first time, and that’s why it’s important to have a feedback loop with your customers.


It’s important to maximize the value of your product. Understanding how your customers are finding value and optimizing their journey so that they can find that value faster.

“Even well-researched products can suffer due to the gap between what customers think they want and what their behaviors reveal they actually want.

By testing a new feature or even a variation of that feature, your product team can see if it is improving the customer experience, or actually making it worse.

The experimentation mindset aligned with the premise of a minimum viable product, seeing how your customers react through a gradual or even incremental roll-out. It allows you to test new ideas before a full launch.

Don’t frame it as a product launch. Just frame it as an experiment.

Dan Siroker
Optimizely Co-Founder and Executive Chairman

If you crave that buzz of a big launch, experimentation after-the-fact enables you to get the most out of those ideas—and your organization’s investment.

By the time something this big has been built, the launch is very, very unlikely to be permanently rolled back no matter what the metrics say.

Carson Forter
Data Scientist at Twitch in his Toward Data Science article, "The Three Types of A/B Tests"

Rather, the randomized experiment, in this case, is for visibility, and to provide information that might help with making future decisions.

Announcing a feature or product through an official launch can inspire adoption—particularly if you have a large customer base to which you can communicate.

But once your product is rolled out, you can start to understand how your customers are interacting with the features at every touchpoint through experimentation.

Product experimentation should make the job of the end user easier or more helpful. If that’s not the goal in your company than you might be running it the wrong way.

Martijn Scheijbeler
Vice President of Marketing at RV.com

Those learnings can be brought forth into future product evolutions—whether that’s changing copy to highlight your value proposition, or smoothing out the funnel flow to reduce friction.

And you can even experiment after your customer logs into your experience, especially when confidentiality is essential for your business.

Do what your customer wants: A painted door experiment example from Optimizely

We created a painted door experiment informing customers we intend to give you this feature, sign up to get early access. We wanted to test if its value proposition resonated with our users,” explains Giannis Psaroudakis of Optimizely.

For those that don’t know–a painted door or a fake door experiment is a way of gauging customer interest in a feature, service or product without building anything.

When the customer clicks a call-to-action button or a link, or even registers with their information, they are notified that it has yet to exist. And they might see something like:

Hey! We haven’t actually built this feature yet, but are you interested?

Of course, you have to know if your customers would be open to this type of experience. You would have to make the experience intriguing enough so that it doesn’t cause frustration.

But you get the data to see if customers find that appealing. Or at least made them curious enough to click or register. And for Optimizely, this worked:

We had several customers enthusiastically send us feedback. It was overwhelming—we received dozens and dozens of requests which is quite uncommon and we even had a customer send us a photo of a thumbs up.

WiderFunnel Experimentation in Product Development Optimizely painted door experiment example
Experimentation is a feedback loop with your customers, but in Optimizely’s painted door experiment, they were able to delight their users with the hope of a new feature.

A painted door experiment, like this, truly exemplifies the power of running a cheap form of experiment as a gauge for the next step.

It gives you the confidence that you’re moving in the right direction, without a single line of code or effort by an engineer.

But more importantly, it’s that feedback loop with your customers.

Relevant resources

The scientific method of product experimentation

Let me first clarify: you don’t test features in product experimentation, you test hypotheses.

Product experimentation takes an optimizer’s brain: It takes an analytical mind to be able to see the opportunities for product experimentation. It also needs the scientific method to turn ideas into measurable hypotheses.

Move away from the mental model of thinking of new products and new features you want to build, in the form of a list of requirements, and instead, think of them as hypotheses.

Giannis Psaroudakis
Director, Product at Optimizely

Rather than saying, “we need to build feature X that has this requirement and that requirement, because customers asked us to”, you start with presenting these same requirements in the form of if/then statements.

This is so important because it sidesteps any HiPPO (highest paid person’s opinion), and it persuades you to think about your product—how you’re building the product—in the form of experiments. In other words, hypotheses that you can validate with data.

Giannis Psaroudakis

Making data-driven decisions is important in product experimentation because the stakes are higher.

You are experimenting within your product experience on your many already acquired customers so you need to have high quality assurance and more guard rails to what makes it to the experimentation stage.

We have a more rigorous review process for product experiment hypotheses,” recalls Giannis Psaroudakis.

We have an experiment review, which is more of an advisory process that allows us to surface new product experimentation ideas and specifically review our hypotheses.

Anyone in the organization can submit their “experiment briefs”—customer challenge, hypotheses, and metrics of success—through Optimizely’s Program Management platform.

And we revise these experiment briefs in the weekly experiment reviews to make sure that whoever is planning to run a product experiment has carefully thought about the hypothesis and the metrics of success.


The scientific method ensures that you can make the data-driven decisions on how your product—and the customer experience—should evolve.

Free Worksheet

Identify opportunities for maximizing your product's value.

When you use the scientific method in experimentation, you don't test features; you test hypotheses. Start planning your first product experiment with our hypothesis worksheet.

Digging deep into data to find that first product experiment opportunity at Acorns

When I joined two years ago, my goal was to improve customer retention,” recalls Hila Qu. “But it is a very broad goal: What do we mean by improving customer retention?

Hila described how she talked to her co-workers—those stakeholders who have been exploring the problem. She asked them the reasons people left their app and their opinions as to what might improve customer engagement.

But she didn’t stop there.

I also worked with our data analytics team to do a quantitative analysis. Basically, I was trying to identify which customer behavior encouraged people to stay.

For example, I looked at the retention curve after a customer completed action A in the product, and compared that to the retention curve of people who didn’t complete action A in the product within their first month.

Using that methodology, I was able to compare different behaviors, because for any digital product, there are a lot of actions user can take.

I was able to narrow down to one particular user behavior where I saw that if the user completes that action, their retention is much better that if the user doesn’t. I also saw that many customers weren’t actually completing that action.

Through her analysis, Hila identified that if a new customer completed action A at the onset of their journey, it had an influence on the customer’s future behavior.

It had a much bigger impact because at that moment, we had people’s attention and they were excited. They wanted to get started with the product.

Hila then mapped out the journey, considering the many paths that a customer could take to complete action A. And there were several:

“For example, you can go to the menu, click that menu item and use that feature; Or, you can basically send them an email with that call-to-action to use that feature. I identified all of the paths so I could understand which one I could focus on first.

So, my hypothesis was based on the data; it seems like if we can get more people to complete action A, the retention will be better. I narrowed down to new customers and if I could get more of them to complete action A, it would be most impactful and easier to influence people’s behavior.

Her first experiment was simple. She didn’t have the development or engineering resources since she was only initiating a product experimentation program.

She hadn’t yet proven experimentation’s value internally, so she needed to identify an opportunity where she could show business impact from a small investment of resources.

We already had a modal in our registration flow that asked people to take action A, but the modal was at the very end of the flow. And you only see the modal if you completed another action. So, I narrowed down to this particular area.

I wanted to test copy. That experiment had the potential for a very high ROI. If it worked, it could have a big impact, but also the effort required was relatively small. It was really just changing the copy and testing different copy against each other.

And it worked.

We launched it and saw over 60% improvement in conversion rate.

Simple or complex—product experimentation can have a big impact.

Further resources

Experimenting broadly and deeply in product development

When it comes to experimentation in product development, you have more opportunity to optimize your experience. That’s because you can experiment broadly and deeply into the product development cycle.

WiderFunnel Experimentation in Product Development Cycle
There are plenty of opportunities to experiment in the product development cycle.

Consider the stages of product development:

First, you have the Discovery stage where you ideate new features or products and this is particularly relevant when you ideate in the form of hypotheses.

According to Meg Watson, Product Manager at Stitch Fix, her team also conducts an analysis to determine if they should proceed into product development.

Next, you move into the Design stage, according to Giannis Psaroudakis, where you can evaluate your hypotheses through early user studies or rapid prototyping. This prototyping is where experimentation begins in terms of usability.

Then, you go on to the Development stage where you dive into coding or testing your actual code.

And afterward, you have the Roll-out stage, when you make sure your hypotheses still stand after launch and there’s nothing negatively affecting the quality of the final product or the customer experience.

In traditional product development, this is where you might stop. Your ideas might win, but they also might fail. And there’s no defined evaluation for where you can improve.

That’s why in product experimentation, you also have the Maximizing stage, that Hila Qu described: What can be improved? What can we evolve? What can make the customer experience frictionless, more delightful?

When I’ve looked at some of the things we’ve started with and usability testing, and then compare that to the product that ships, X months later, a lot happens in between those stages. We start with sending a prototype through usability and then we have all the different optimization tests that run after something goes live.” confirms Meg Watson.

Throughout the product development cycle, with the experimentation mindset, you are constantly exploring and validating the possibilities for new features or products by generating that feedback loop with your customers.

But perhaps more importantly, experimentation in product development means you can test deep in your stack.

Unlike client-side experimentation, with its what-you-see-is-what-you-get approach, you can test what you can’t see. Product experimentation incorporates server-side testing:

You can test changes on your server side and improve the performance of your back-end systems,” Giannis Psaroudakis affirms.

If you’re running any machine learning models that give recommendations to customers, you can experiment with those, for instance.

In client-side experimentation, a variation is run through the web browser or mobile app using a visual editor and a single JavaScript line. Once the variation is proven, you can build it out by implementing a code pack.

But in server-side experimentation, the variation is coded on servers. So, it can be more resource intensive because you build up-front.

WiderFunnel Experimentation in product development client-side server-side
You can see that server-side experimentation requires that you build upfront. (Source: “Why Experiment Server-Side?” by Optimizely)

That way, you can experiment with how a product functions, like the machine learning algorithms that enables you to deliver a personalized experience.

And that’s why product experimentation has so much potential.

Learn more from these innovative organizations

When it comes to experimentation, it’s all about the customer experience.

Ultimately the goal of all this, the reason we’re doing this, is to give the customer a better experience with Stitch Fix and to make sure that it’s effortless. It’s fun. it’s delightful. And that they truly have a good experience with all of our products.

Meg Watson

Experimentation in product development allows you to get deeper into the stack, deeper into your customer’s experience so that you are delivering the best possible solution and a delightful one at that.

For many organizations, it’s an untapped source of growth.

The gap between product and marketing teams is becoming smaller and smaller, and from a customer experience perspective, it’s blended.

Hila Qu

The customer experience has to be consistent and compelling, and work seamlessly between the pieces that are controlled by the two teams, in order to have the customer finish that journey from a random visit or from your ads, to the point where they use the product for the first time and they know the value, they see the benefit.

WiderFunnel Product and Marketing Customer Experience
If the customer doesn’t view marketing and product as separate parts of their journey, organizations shouldn’t either.

But we need to stop siloing product and marketing.

With an experimentation mindset, organizations need to unite internally to spread those insights throughout every customer touchpoint.

That means experimenting constantly—both client-side and server-side—and sharing the insights across both marketing and product development teams.

Because your customers don’t see your landing page or your product funnel as distinct experiences. Instead, you need to focus on the entire customer journey.

What are your burning product experimentation questions? We’d love to hear them below.

Author

Lindsay Kwan

Marketing Communications Specialist

Grab your complete copy of the new “State of Experimentation Maturity 2018” research report

What makes some organizations so successful when it comes to experimentation? This new 45-page report provides benchmarks for stages of experimentation maturity at leading North American brands.

Get Report