Welcome back to another edition of the Widerfunnel Experimentation News Roundup! These are the headlines that caught our attention over the last month, along with upcoming events and job postings you may be interested in. Did we miss something major? Let us know in the comments!
Experimentation News: October 2019
Why eRetailers Should Turn to Experimentation Rather Than Surge Moments to Boost Business Growth – MarTechSeries.com – by Carl Tsukahara
Digital Experience Optimization, or DXO, is the more stable, strategic foil to surge moments, which are just that – high-pressure shopping blitzes. In contrast, DXO empowers businesses to reliably plan for growth throughout the year by facilitating constant experimentation and iteration to land on digital experiences that resonate with customers and inspire them to engage time and time again with the brand. This sustained, always-on testing provides a constant influx of data that allows the business to course-correct at a faster pace than they would with one standalone moment in time.
Mature companies outside of the tech space do not naturally experiment, learn and pivot. It is absolutely critical that these companies develop this muscle, as it’s this skill deficit that’s causing them to struggle, according to Macy’s Chief Digital and Revenue Officer Jill Ramsey.
In this new report published September 2019, “Harnessing the experimentation mindset to drive growth”, we’ve gathered new research data from decision-makers in Digital, Marketing, and Product from leading brands across multiple industries including Microsoft, Target, Allstate, Domino’s, BMW, and more, to uncover the characteristics of a healthy culture of experimentation.
The Good, The Bad and The Testable: How to Give Your Experiment the Best Foundation – DevOps.com – by Lizze Eardley
- How Do You Spot a Bad Metric or Wrong Test?
- Consequences of a Bad Metric or Wrong Test
- What Is a Good Metric?
- Benefits of a Good Metric
Experimentation and Startup Performance: Evidence from A/B Testing – Harvard Business School – By Rembrand Koning, Sharique Hasan, and Aaron Chatterji
Is experimentation the right strategy for startups? This analysis of the adoption of A/B testing technology by 35,000 global startups provides evidence that a strategy based on repeated experimentation will improve performance over time. However, the benefits of experimentation vary. Experimentation helps younger startups “fail faster,” while older firms may discover new, high-growth products.
YouTube is experimenting with ways to make its algorithm even more addictive – TechnologyReview.com – By Karen Hao
Among the proposed updates, the researchers specifically target a problem they identify as “implicit bias.” It refers to the way recommendations themselves can affect user behavior, making it hard to decipher whether you clicked on a video because you liked it or because it was highly recommended. The effect is that over time, the system can push users further and further away from the videos they actually want to watch.
To reduce this bias, the researchers suggest a tweak to the algorithm. When the researchers tested the changes live on YouTube, they found significantly more user engagement.
Facebook Is Experimenting With Artificial Intelligence That Let’s You Digitally Get Dressed – TheSource.com – By Keenan Higgins
Facebook believes that Fashion++ will change the game of existing digital style guide apps by helping you tweak your existing outfit instead of shopping and spending on more gear.
Mastercard experimenting with palm print biometrics on smartphones in digital shift – BiometricUpdate.com – By Chris Burt
Mastercard showed off its biometric smart cards at the event, and also its experiments with biometric authentication using a photograph of the user’s palm taken on a smartphone. Because it does not rely on a dedicated hardware feature, Mastercard believes it could be more widely accessible than fingerprint authentication.
United’s MilePlay program, which offers travel incentives via targeted promotions to members of the airline’s MileagePlus loyalty program, reflects the company’s overall digital experimentation with respect to machine learning.
Creating a multi-channel approach to experimentation is less about getting it right up front – you’ll learn, evolve, and change things up along the way. What’s critical is that you explore the opportunities to add value and group them in some fashion. Beyond the lab we leveraged opportunities to engage the university (professors, students, etc.), connect with other co-located labs, and reach a sophisticated start-up community.
“A digital subscription renaissance”: Highlights from Shorenstein Center’s Digital Pay-Meter Playbook – What’s New In Publishing/Medium.com – Team WNiP
Highlights from the whitepaper:
- Publishers with over 6% stop rates have “thriving” digital subscriptions businesses
- Best performing digital publishers have 3x engagement rates
- Subscriber churn compounds over time
Indian enterprises need mindset of experimentation: Amit Gautam, CDO, NLBS – CIO.com – by Bhragu Haritas
What best practices can Indian companies adopt from their US counterparts?
Probably the most important is the mindset of experimentation. It varies heavily based on various industries and their adoption of technology but I think the most important thing Indian enterprises can learn is to start experimenting and investing into infrastructure that can give them the edge for tomorrow. Building the architecture and infrastructure, gathering more data points and information, and then finding signals from them are some of the things Indian enterprises can do better.
There are now more options in the “Allow game captures” setting. You can choose between “Captures by me,” “Captures by me or games,” or “Don’t capture.” Additionally, there is a new experimental update called “Capture & Share” that will only be provided to a “limited portion of the Xbox One audience to gauge interest and collect feedback.” Yep, they’re doing A/B testing in production code now.
“Software feature experimentation tips with users or in production may sound scary and complicated, but it doesn’t have to be. These feature experimentation tips can help you determine if experimentation is right for you.”
The Cloud Is Transforming How We Build Software — Here is How Companies Can Prepare – Forbes – By Eugene Khazin
The purpose of building a cloud-native data platform is to make data a first-class citizen. This means that all critical data within an organization is not only trustworthy but accessible to others within an organization.
When data is in the right format, trustworthy and accessible, this feeds the success of digital transformations. Domino’s is a great example of a data-led digital transformation. They used A/B testing against every new tool and process to optimize their results.
I hope you’ll find this fairly detailed overview and solutions for the 5 most common mistakes in using statistical methods in online A/B testing useful and practical. These are costly mistakes I see in my daily practice both as a consultant and a tool vendor. I really think we should see much fewer instances of these, especially given how easy most of them are to avoid.
Like AI systems such as Alexa and Deep Cube, the proficiency of any digital marketing campaign will be based on how quickly it can intelligently react to outside stimuli and re-adapt its messaging and targeting process (this is called A/B testing) to drive the greatest results in the law of averages.
Once your marketing system is running at the highest efficiency — which we marketers term product-market fit — you’re then in a position to scale in a streamlined, reliable manner.
Upcoming experimentation events and webinars
Past events and webinars
Have an experimentation event coming up? Please email [email protected] so I can include it in the next edition!
Thomson Reuters – Sr. Digital Testing and Optimization Manager (Carrollton, TX)
Agoda – Head of Experimentation (Seattle,WA)
Avis Budget – Director, Insights & Analytics (New Jersey)
Credit Karma Senior Growth Marketing Manager (San Francisco, CA)
Twitter – Senior Product Manager – Product Insights & Experimentation (San Francisco, CA)
Facebook- Content Strategy Manager, New Product Experimentation (Menlo Park, CA)
Hulu – Senior Software Developer – Experimentation (Seattle, WA)
Hulu – Experimentation Manager (Santa Monica, CA)
Do you know of an upcoming event or current job posting we should include in this round-up? Let us know in the comments section below, and stay tuned for our next experimentation news round-up in two weeks!