March 2020 round-up: Culture of experimentation, “experimentation works” book, selecting a testing tool, and more!

By: Daniel Molina
11 min. read Last updated: August 26th, 2020

Widerfunnel Experimentation News Round-Up for March is here! There’s been a lot of content around experimentation lately, and I think a lot of it has to do with the new book by Stefan Thomke titled “Experimentation Works: The Surprising Power of Business Experiments” paired with HBR’s front cover for March/April 2020 edition (first link below). Here’s a few headlines from the past month that touch on this topic, along with a snippet from the each article.


experimentation news round-up articles

Experimentation News: March 2020

Building a Culture of Experimentation – Harvard Business Review – by Stefan Thomke

“In an increasingly digital world, if you don’t do large-scale experimentation, in the long term—and in many industries the short term—you’re dead,” Mark Okerstrom, the CEO of Expedia Group told me. “At any one time we’re running hundreds, if not thousands, of concurrent experiments, involving millions of visitors. Because of this, we don’t have to guess what customers want; we have the ability to run the most massive ‘customer surveys’ that exist, again and again, to have them tell us what they want.” Read more…

Big Tech Is Testing You – New Yorker – By Hannah Fry

In fact, if you’ve recently used Facebook, browsed Netflix, or run a Google search, you have almost certainly participated in an experiment of some kind. Google alone ran fifteen thousand of them in 2018, involving countless unsuspecting Internet users. “We don’t want high-level executives discussing whether a blue background or a yellow background will lead to more ad clicks,” Hal Varian, Google’s chief economist, tells the authors. “Why debate this point, since we can simply run an experiment to find out?” Read more…

A Smarter Way to Run Business Experiments – Marker – By Stefan Thomke

Many organizations are reluctant to fund good business experiments and have considerable difficulty executing them. Most tests of new business initiatives are too informal. They are not based on proven scientific and statistical methods, and so executives end up misinterpreting statistical noise as causation — and make bad decisions. To ensure that each experiment yields better decisions — companies should ask themselves seven important questions. Read more… 

What The Best Horror Films Can Teach Us About Website Experience Design – Forbes – by Chris Goward

That’s one reason we often see redesigned experiences hurting conversion performance even if they’re objectively better. People are simply accustomed to the old, worse one. This makes overhaul redesigns risky! The best way to avoid the redesign risk is to iteratively experiment your way to a new design. Not only will you make design changes more incrementally so users become used to them bit by bit; you’ll also reduce risk by proving which individual changes are improvements. Read more…

Have you made it safe to fail? – McKinsey Digital – Book Excerpt

Making it safe to fail is crucial because learning happens through experimentation, and experimentation often results in failure. Recent McKinsey research shows, in fact, that respondents at successful organizations are more than twice as likely as their peers elsewhere to strongly agree that employees are rewarded for taking risks of an appropriate level.2 So a willingness to fail has to be embedded in a company’s culture.  Read more…

Why A Culture Of Experimentation Requires Management Transformation –  Forbes – By Steve Denning 

Creating a culture of experimentation involves more than adding a fix on top of the static machine with a vertical reporting dynamic. The whole organization needs to become an organic living network of high-performance teams. In such firms, managers recognize that competence resides throughout the organization and that innovation can and must come from anywhere. In an Agile organization, continuous experimentation is the normal way of life. Read more…

6 Features to look for in an enterprise testing tool – Widerfunnel Blog – By Alice Liang

For whatever reason when people are choosing an expensive tool, they are expecting it to be easier to operate because they are paying more for it. But it is really the opposite. A Lamborghini is much more difficult to drive than a Civic. Expensive tools should be operated by experts, so honestly evaluate the skills of your team to ensure you will get the most out of your investment. Read more…

‘Experimentation Works’ and ‘The Power of Experiments’ Review: Test, Test and Test Again – Wall Street Journal – By David A. Shaywitz

The application of this method to business has gained new urgency with the explosion of online commerce and the recognition that small tweaks can generate enormous profits. As Stefan Thomke notes in “Experimentation Works,” tests at Microsoft in 2012 revealed that a tiny adjustment in the way its Bing search engine displayed ad headlines resulted in a 12% increase inrevenue, translating into an extra $100 million annually for the company in the U.S. alone. Read more…

Highlighting the link between digital experimentation and growth – Kameleoon Blog – By Anne-Claire Bellec

What are the financial benefits of digital experimentation? To find out, global experimentation consultancy GO Group Digital benchmarked brands across the world, looking at the link between growth and the use of experimentation. We interviewed Chris Goward of Widerfunnel, one of GO Group Digital’s partner agencies, about the findings and what they mean for brands. Read more…

Experimentation Works: The Surprising Power of Business Experiments – Optimizely Blog – By Todd Krieger 

Professor Thomke has been pioneering in the field of experimentation since before Optimizely was founded, with his first book on the topic, ‘Experimentation Matters: Unlocking the Potential of New Technologies for Innovation‘ coming out in 2003.

His new book ‘Experimentation Works: The Surprising Power of Business Experiments‘ was recently published and in addition to collaborating with him, we had the chance to sit down with him and discuss some of his findings. Read more…

Why India is the newest hotbed for tech experimentation – – By Shubham Agarwal

Finding a foothold in India — which has a multitude of regional languages, unique customer behaviors, and a relatively nascent internet population — isn’t exactly a walk in the park for these companies. Often, success in the West doesn’t translate well in the country. That’s where the experimentation comes in. Read more…


experimentation news roundup - upcoming events

Past Events and Webinars from Widerfunnel

[NEXT WEBINAR] The Surprising Science of Emotion Behind Fast Growth – March 18th 2020 – Widerfunnel

Past webinars:

experimentation news round-up articles

Who’s hiring?

Do you know of an upcoming event or current job posting we should include in this round-up? Let us know in the comments section below, and stay tuned for our next experimentation news round-up next month!

Bonus Meme:

some image

Discover how your experimentation program stacks up!

Benchmark your experimentation maturity with our new 7-minute maturity assessment and get proven strategies to develop an insight-driving growth machine.

Get Started