October 2021: Experimentation News Round Up

Experimentation News

The Art of Possible – A Disney Technology Blog

Universal Holdout Groups at Disney Streaming | By Tian Yang

In this blog post, the Disney Streaming team talks about their experience piloting “holdout groups” for Hulu subscribers. Learn more about what a holdout group is, why they needed it and the lessons learned.

Interpreting A/B test results: false positives and statistical significance

Netflix Technology Blog | By Martin Tingley with Wenjing Zheng, Simon Ejdemyr, Stephanie Lane, and Colin McFarland

Experimentation team at Netflix continues their multi-part series on how Netflix uses A/B testing to inform decisions and innovate. This is Part 3, where they focus on “building intuition” of what a false positive is.

Interpreting A/B test results: false negatives and power

Netflix Technology Blog | By Martin Tingley with Wenjing Zheng, Simon Ejdemyr, Stephanie Lane, and Colin McFarland

This is part 4 of the multi-part series on A/B Testing by the Netflix experimentation team, focusing on false negatives and statistical power.

Behavioural Science in the Sign-up Flow

Gousto Engineering & Data Blog | By Kai Mackie

In this blog post, Kai Mackie, Product Analyst at Gousto, shares how Behavioral Science plays a role in a/b testing at Gousto. Some real-world examples include leveraging social proof and loss aversion.

Personalisation journey: going from a large to small team

Adevinta Tech Blog | By Jordi Mas

Jordi Mas, Engineering Manager at Adevinta, shares his experience with scaling down the team to operate more efficiently, using a data-driven approach. Follow their story on how they took a team of 14 and divided it into smaller teams, leveraging A/B testing to compare results. Adevinta seems to have a very strong culture of experimentation, where they apply the same mindset to overcome their own internal operational challenges.

How A/B testing can (and can’t) improve your headline writing

NiemenLab | By Nick Hagar and Nicholas Diakopolous

Many news organizations like the New York Times or The Boston Globe are A/B testing their headlines. So NiemanLab partnered with Chartbeat, an analytics tool that offers A/B testing, to analyze 141,000 headline A/B tests run by 293 news websites. Find out what they learned.

Unlocking digital potential through experimentation and personalisation to drive better outcomes

New Digital Age | By Justin Anovic

The Chief Product Officer of Optimizely, Justin Anovic, shares his perspective and advice on personalization and experimentation. Includes a story on how AeroMexico was able to personalize their check-out experience to boost conversion rates

Optimizely Survey: Consumers Report Heightened Expectations for Digital Experiences this Holiday Season

Yahoo! Finance | By Business Wire

Optimizely revealed results from a new survey of 5,000 consumers in the US, UK, Germany, Australia and Sweden. Some of the findings include that, on average, American consumers expect their holiday spending budgets to increase 25% compared to last year. Great news for retailers!