< Back to main blog page

November 2021: Experimentation News Round Up

Words by: Daniel Molina Last updated: December 1st, 2021 13 min. read

  • Experimentation News

Experimentation News

Tracking Regret, the Cost of Bad Decisions

Change.org Medium Blog | By Dimitri Tishchenko

“At Change.org, we rely on the decision making of hundreds of thousands of autonomous learning agents (a.k.a mutli-armed bandits) to select the right headline for each petition per share channel. This article describes how we quantify and evaluate the efficiency with which these agents learn and perform.”

Read more

Building confidence in a decision

Netflix Technology Blog | By Martin Tingley with Wenjing Zheng, Simon Ejdemyr, Stephanie Lane, Michael Lindon, and Colin McFarland

“This is the fifth post in a multi-part series on how Netflix uses A/B tests to inform decisions and continuously innovate on our products. Here, we’ll get to the hard part: how do we use test results to support decision making in a complex business environment?”

Read more

Stop Making Bad Product Decisions — Start Applying Continuous Experimentation

Personal Blog on Medium.com | By Stefan Berreiter

“This is the first post in a multi-part series covering my ongoing research in the area of ‘Continuous Experimentation in Live Software Environments’. In this episode, I talk about what Continuous Experimentation (CEx) is, why it exists, and which strategies you can directly apply. Episode two summarizes key learnings from 15+ SaaS companies on their CEx activities.”

Read more

Parameter Exploration at Lyft

Lyft Engineering Blog | By Henry Quan

At Lyft, the experimentation team’s purpose is to drive data-driven decision making. In this blog post, Henry explains what parameters are, how they can be optimized, and the challenges they faced.

Read more

Randomization, Blocking, and Re-Randomization: Three fundamental pillars for online experimentation

TowardsDataScience.com Blog | By Leihua Ye, Ph.D. Researcher

“In today’s blog post, let’s discuss the three fundamental statistical concepts that contribute to the high validity of Online Experimentation: randomization, blocking, and re-randomization.”

Read more

Leaders: Stop Confusing Correlation with Causation

Harvard Business Review | By Michael Luca

“We’ve all been told that correlation does not imply causation. Yet many business leaders, elected officials, and media outlets still make causal claims based on misleading correlations. These claims are too often unscrutinized, amplified, and mistakenly used to guide decisions.”

Read more

How to set the Minimum Detectable Effect in AB-Tests

TowardsDataScience.com Blog | By Dennis Meisner

“Determining a Minimum Detectable Effect (MDE) value is one of the trickier parts whenever setting up an AB-Test with product teams. There exists a lot of confusion about what this term means. And about what value should be chosen for a particular test. In this article, I want to shed some light on the meaning of the MDE, its impact on the experiment results, and how to find an appropriate value.”

Read more

Researchers Amp up Digital Experimentation

MIT Digital Medium Blog | By Paula Klein

“The convergence of online experimentation and behavioral science was the focus of this year’s MIT Conference on Digital Experimentation (CoDE). The annual two-day convocation featured eight plenary speakers and 100 presentations. Many offered methods that can scale to hundreds of thousands of people and be used by policy makers to address problems from COVID vaccination reluctance to online marketing preferences.”

Read more

Data Quality: Fundamental Building Blocks for Trustworthy A/B testing Analysis

Microsoft ExP Blog | By Platina Liu, Wen Qin, Hao Ai, Jing Jin

“In this blog post, we will explore tools and methodologies that we can use to ensure that we are relying on quality data to produce trustworthy A/B testing analyses. We will answer the following questions: How can data quality impact our A/B testing results? What are the critical data quality requirements for A/B testing? How can we continuously monitor data quality?”

Read more

Speeding up A/B Tests with Delayed Reward Forecasting

Wayfair Tech Blog | By Le Wang

“Some experiments at Wayfair can last 60 days or more. To speed up learning in experiments while still optimizing for long term rewards, our team developed a data science platform called Demeter, that uses ML models to forecast longer term KPIs based on customer activity in the short term. In this post we provide an overview for Demeter and its theoretical foundation in causal inference.”

Read more

Improving Subgroup Analysis with Stein Shrinkage

DoorDash Engineering Blog | By David Kastelman

“When estimating the treatment effect for different segments, variance poses a number of issues. It increases error size and can lead to a misleading ordering of the most affected segments, with smaller segments exhibiting more extreme effects because of their larger variances. To address these issues at DoorDash while optimizing promotions, we had notable success using Stein shrinkage.”

Read more

Who’s Hiring?

Design Lead| WiderfunnelVancouver, BC

Digital Optimization Strategist| WiderfunnelVancouver, BC

Digital Programs Manager| WiderfunnelVancouver, BC

Conversion Rate Optimization Specialist| The Motley FoolUSA

Ecom Growth Optimization Manager| adidasPortland, OR

Senior Backend Engineer, Growth: Experimentation| GitLabVancouver, BC (Remote)

Conversion Optimization Manager| ProperlyCanada (Remote)

Senior Manager, Conversion Experimentation| Disney StreamingNYC, NY

Director, Growth Marketing| RealmNYC, NY

Specialist, Digital Optimization| WileyRemote, USA

Software Engineering Manager – Experimentation Platform| Sony Interactive Entertainment PlayStation San Francisco, CA

Senior Software Engineer – Experimentation| Realtor.comVancouver, BC

Sr. Marketing Manager, Experiments| AmazonSeattle, WA

Senior Product Manager, Experimentation| ZipRecruiterAustin, TX

Director of Experimentation Engineering| NikeBeaverton, OR

Experimentation Lead, Web Presence| StripeNew York, NY

Experimentation Advocate| RobinhoodMenlo Park, CA

Lead, Experimentation| Macy’sNYC, NY

Sr. Product Manager, Experimentation| WarnerMediaCulver City, CA

Director, Experimentation| ComcastPhiladelphia, PA

Optimization Manager| FenderLos Angeles, CA

Subscribe to get experimentation insights straight to your inbox.