< Back to main blog page

January 2022: Experimentation News Round Up

Words by: Daniel Molina Last updated: January 27th, 2022 11 min. read

  • Experimentation News

Experimentation News

Disney Streaming Embraces 3 Key Tenets of Experimentation

Disney Streaming Blog | By Diana Jerman, Mark Harrison, Anmeen Leong, Michael Ramm

“At Disney, experimentation has played a big role in our ever-evolving history and goal of delighting customers. From experiments in animation, storytelling, and robotics, we encourage our teams to look at challenges as opportunities to invent and innovate.”

Read more

Experiment without the wait: Speeding up the iteration cycle with Offline Replay Experimentation

Pinterest Engineering Blog | By Maxine Qian

“Could we predict experiment outcomes without even running an experiment? Could it be done in hours instead of weeks? Could we rapidly pick only the best ideas to run an online experiment? This post will describe how Pinterest uses offline replay experimentation to predict experiment results in advance.”

Read more

Improving Duolingo, one experiment at a time

Duolingo Blog | By Lavanya Aprameya

“Running as many experiments as we do all at once doesn’t come without caveats, though. A big one is determining how to best gather and synthesize the data points that emerge from each experiment, which in turn gives us a better understanding of which changes we should (or shouldn’t) make.”

Read more

Multiple Comparison: A Common Pitfall for A/B Testing

TowardsDataScience.com Blog | By Leihua Ye, Ph.D. Researcher

“At Change.org, we rely on the decision making of hundreds of thousands of autonomous learning agents (a.k.a mutli-armed bandits) to select the right headline for each petition per share channel. This article describes how we quantify and evaluate the efficiency with which these agents learn and perform.”

Read more

Netflix: Lessons in Experimentation

Product Growth Blog | by Aakash Gupta

“So, for today’s piece, I am excited to present a collaboration with my friend Bandan, a product leader at Booking.com ($100B market cap), and before that Gojek (valued at $30B). We have been doing quite a bit of research into Netflix’s experimentation.”

Read more

Why It Matters Where You Randomize Users in A/B Experiments

Medium.com Blog @foundinblank | By Adam Stone

“A common experimentation design decision is which population to randomize. Should you randomize all users that could theoretically be exposed to a new feature, or only users who definitely would be exposed? At the top of the funnel, or somewhere halfway through the funnel?”

Read more

Improving Duolingo, one experiment at a time

Duolingo Blog | By Lavanya Aprameya

“Running as many experiments as we do all at once doesn’t come without caveats, though. A big one is determining how to best gather and synthesize the data points that emerge from each experiment, which in turn gives us a better understanding of which changes we should (or shouldn’t) make.”

Read more

Experimentation is a major focus of Data Science across Netflix

Netflix Technology Blog | By Martin Tingley with Wenjing Zheng, Simon Ejdemyr, Stephanie Lane, Colin McFarland, Andy Rhines, Sophia Liu, Mihir Tendulkar, Kevin Mercurio, Veronica Hannan, Ting-Po Lee

“Here we describe the role of Experimentation and A/B testing within the larger Data Science and Engineering organization at Netflix, including how our platform investments support running tests at scale while enabling innovation.”

Read more

Netflix: A Culture of Learning

Netflix Technology Blog | By Martin Tingley with Wenjing Zheng, Simon Ejdemyr, Stephanie Lane, Colin McFarland, Mihir Tendulkar, and Travis Brooks

“The secret sauce that turns the raw ingredients of experimentation into supercharged product innovation is culture. There are never any shortcuts when developing and growing culture, and fostering a culture of experimentation is no exception.”

Read more

Novelty Bias in Experimentation

Rebin Hosini Medium Blog | By Rebin Hosini

“Having feature adoption is often considered a positive trait, but can sometimes lead to misinterpretation. Part of the ambiguity can be attributed to the short term effect of releasing a new product/feature i.e novelty bias.”

Read more

Growth Experiments Prioritization

Guilherme Lacerda Medium Blog | By Guilherme Lacerda

“These days I got a message from a PM colleague through Slack, right after I presented the results for our latest Experiment. He wanted to know what my team was doing right in terms of prioritization since we were getting good results from our recent tests.”

Read more

Experimentation-driven development

StatSig Blog | By Ritvik Mishra

“I worked on Facebook News Feed before I joined Statsig, and that’s where I learned about the value of experimentation. The culture in News Feed wasn’t just to use experiments to verify that our intuitions about what changes may lead to improvements were true.”

Read more

Who’s Hiring?

Website Product Owner| Best EggWilmington, DE

Digital Optimization Strategist| WiderfunnelVancouver, BC

Digital Programs Manager| WiderfunnelVancouver, BC

Manager, Experimentation| Canadian Tire CorpOntario, CA

Ecom Growth Optimization Manager| adidasPortland, OR

Director, Growth Marketing| RealmNYC, NY

Sr. Marketing Manager, Experiments| AmazonSeattle, WA

Experimentation Lead, Web Presence| StripeNew York, NY

Lead, Experimentation| Macy’sNYC, NY

Sr. Product Manager, Experimentation| WarnerMediaCulver City, CA

Conversion Rate Optimization Manager| SquareSan Francisco, CA Remote

Conversion Rate Optimization Manager| PearsonMiami, FL

Conversion Rate Optimization Specialist| General MotorsDetroit, MI

Growth Marketing, Website Optimization Lead – HelloSign| DropboxSeattle, WA (Remote, Flexible location)

Subscribe to get experimentation insights straight to your inbox.