April 2022: Experimentation News Round Up

The Fine Art of Visualizing Experiment Results

Disney Streaming Technology Blog | By Allen Chang

In this post from the Disney Streaming technology blog, Allen Chang (Sr. Data Scientist, Core Experimentation) shares how they help teams understand the results of their experiments through data visualization. The post covers the specific visualizations they use to represent statistical uncertainty as well as the tech that they use to build them.

Musa Tariq, GoFundMe's CMO, is testing how to bring people together.

Forbes | By Jair Hilburn

In this Forbes article, Musa Tariq (CEO of GoFundMe) explains the value and importance of testing. Musa understands that most experiments may fail, but “knowing that if one of those things work, it could be the thing that changes both the trajectory of the company and therefore helps more people.”

How Netflix tests Netflix: The story behind the service’s new two-thumbs-up feature

Protocol | By Janko Roettgers

Detailed story about how Netflix landed on the two-thumbs-up feature and the influence of a/b testing and customer research when making that decision.

Using Experimentation To Understand Tradeoffs In Referral Incentives

GrowthHackers Community Blog | By Ilya Izrailevsky

In this post, Ilya Izrailevsky (Senior Engineering Manager, Experimentation & Metrics at Robinhood) shares what the Robinhood experimentation team discovered about the effect of referral incentives on a product, and some of the approaches to developing it.

Act Like a Scientist

Harvard Business Review (May-June 2022 Magazine) | By Stefan Thomke and Gary W. Loveman

“Drawing on the experiences of Harrah’s Entertainment, Sony, Bank of America, and Lego, Thomke and Loveman show how scientific methods can help companies discard ineffective practices, increase marketing and operational efficiency, boost customer satisfaction and sales, find new sources of growth, and even turn around struggling businesses.”

Challenges in Experimentation

Lyft Engineering Blog | By John Kirn

In this post from Lyft’s engineering blog, John Kirn (Product Manager for Experimentation) shares where Lyft’s largest challenges come from and how it’s the experimentation team’s job to solve these challenges. The post then focuses on the capabilities they have deployed in order to “enable decision-making” at Lyft as well as their plans for the future.

Overtracking and trigger analysis: reducing sample sizes while INCREASING the sensitivity of experiments

Booking.com’s Data Science Blog | By Pablo Estevez and Or Levkovich

In this post from Booking.com’s Data Science Blog, Pablo Estevez (Head of Data Science & ML) and Or Levkovich (ML Scientist) share their thoughts on “overtracking” and how correcting overtracking can be one of the “most cost-efficient ways to reduce the minimum detectable effect of experiments, all while reducing sample sizes and subject-recruitment times.”

Even Split Increases Power of A/B Tests

Towards Data Science Blog | By Qike (Max) Li

In this post published on the Towards Data Science blog, Qike (Max) Li (Data Scientist at Wish), shares their learnings when setting up the experiment sample ratio to reach statistical significance faster.