Experimentation News
Disney Streaming Embraces 3 Key Tenets of Experimentation
“At Disney, experimentation has played a big role in our ever-evolving history and goal of delighting customers. From experiments in animation, storytelling, and robotics, we encourage our teams to look at challenges as opportunities to invent and innovate.”
Experiment without the wait: Speeding up the iteration cycle with Offline Replay Experimentation
“Could we predict experiment outcomes without even running an experiment? Could it be done in hours instead of weeks? Could we rapidly pick only the best ideas to run an online experiment? This post will describe how Pinterest uses offline replay experimentation to predict experiment results in advance.”
Improving Duolingo, one experiment at a time
“Running as many experiments as we do all at once doesn’t come without caveats, though. A big one is determining how to best gather and synthesize the data points that emerge from each experiment, which in turn gives us a better understanding of which changes we should (or shouldn’t) make.”
Multiple Comparison: A Common Pitfall for A/B Testing
“At Change.org, we rely on the decision making of hundreds of thousands of autonomous learning agents (a.k.a mutli-armed bandits) to select the right headline for each petition per share channel. This article describes how we quantify and evaluate the efficiency with which these agents learn and perform.”
Netflix: Lessons in Experimentation
“So, for today’s piece, I am excited to present a collaboration with my friend Bandan, a product leader at Booking.com ($100B market cap), and before that Gojek (valued at $30B). We have been doing quite a bit of research into Netflix’s experimentation.”
Why It Matters Where You Randomize Users in A/B Experiments
“A common experimentation design decision is which population to randomize. Should you randomize all users that could theoretically be exposed to a new feature, or only users who definitely would be exposed? At the top of the funnel, or somewhere halfway through the funnel?”
Improving Duolingo, one experiment at a time
“Running as many experiments as we do all at once doesn’t come without caveats, though. A big one is determining how to best gather and synthesize the data points that emerge from each experiment, which in turn gives us a better understanding of which changes we should (or shouldn’t) make.”
Experimentation is a major focus of Data Science across Netflix
“Here we describe the role of Experimentation and A/B testing within the larger Data Science and Engineering organization at Netflix, including how our platform investments support running tests at scale while enabling innovation.”
Netflix: A Culture of Learning
“The secret sauce that turns the raw ingredients of experimentation into supercharged product innovation is culture. There are never any shortcuts when developing and growing culture, and fostering a culture of experimentation is no exception.”
Novelty Bias in Experimentation
“Having feature adoption is often considered a positive trait, but can sometimes lead to misinterpretation. Part of the ambiguity can be attributed to the short term effect of releasing a new product/feature i.e novelty bias.”
Growth Experiments Prioritization
“These days I got a message from a PM colleague through Slack, right after I presented the results for our latest Experiment. He wanted to know what my team was doing right in terms of prioritization since we were getting good results from our recent tests.”
Experimentation-driven development
“I worked on Facebook News Feed before I joined Statsig, and that’s where I learned about the value of experimentation. The culture in News Feed wasn’t just to use experiments to verify that our intuitions about what changes may lead to improvements were true.”
Who’s Hiring?
Website Product Owner| Best Egg – Wilmington, DE
Digital Optimization Strategist| Widerfunnel – Vancouver, BC
Digital Programs Manager| Widerfunnel – Vancouver, BC
Manager, Experimentation| Canadian Tire Corp – Ontario, CA
Ecom Growth Optimization Manager| adidas – Portland, OR
Director, Growth Marketing| Realm – NYC, NY
Senior Analyst, AB Testing and Data Operations| 2U – NYC, NY
Sr. Marketing Manager, Experiments| Amazon – Seattle, WA
Experimentation Lead, Web Presence| Stripe – New York, NY
Lead, Experimentation| Macy’s – NYC, NY
Sr. Product Manager, Experimentation| WarnerMedia – Culver City, CA
Senior Data Scientist, Experimentation| Reddit – Remote
Product Analytics and Experimentation Manager| Calendly – Remote, USA
Conversion Rate Optimization (CRO) Manager| WW – Washington, USA
Conversion Rate Optimization Manager| Square – San Francisco, CA Remote
Conversion Rate Optimization Manager| Pearson – Miami, FL
Conversion Rate Optimization Specialist| General Motors – Detroit, MI
Growth Marketing, Website Optimization Lead – HelloSign| Dropbox – Seattle, WA (Remote, Flexible location)
Subscribe to get experimentation insights straight to your inbox.