Experimentation news round-up #4: Dark patterns, insights from Bezos, and learnings from 20,000+ experiments

By: Chris Goward
9 min. read Last updated: August 27th, 2020

Experimentation news, events, and careers

Welcome back to another experimentation news round up from WiderFunnel. If you missed the previous three round-ups, you can check them out here.

If you’re a first-time reader, the round-up is a collection of the latest experimentation-focused articles, industry news, events and webinars, and relevant job postings.

experimentation news round-up articles

Experimentation news: July / August

Here are the headlines we’ve been paying attention to the last couple of weeks…

Running Your Business Should Be An Everlasting Experiment – Forbes

Shawn Freeman, Founder of TWT explains via Forbes that “Running a business is very much like the scientific method, starting with a question, research and a hypothesis before you test the hypothesis, analyze the data and draw conclusions. If the results don’t align with your initial hypothesis, you either learn from the results or you go back to the lab, build a new theory and start the process over again.”

6 ways Big Tech is tricking you: A guide to recognizing Dark Patterns – Fast Company

With The Detour Act (which would make so-called “dark patterns” illegal) working its way through Congress, Fast Company lists examples of what UX Researcher Harry Brignull coined “dark patterns” back in 2012—practices organizations employ that are considered to be deceptive by design.

Learn which companies have been accused of using tactics like Misdirection, Disguised Ads and Confirm Shaming and how these tactics work.

Social media experimentation key to online success, says man who caught Elon Musk’s eye with world-famous ‘Absolute Unit’ – Attractions Management

I wanted to include this story because it captures the spirit of getting out of your comfort zone when it comes to experimentation. It covers Adam Koszary, the social media manager at the Museum for English Rural Life, who managed to grow the museum’s twitter following from 9,700 to 134,000 in under a year.

By taking an out-of-the-box approach to content from an established museum, and eventually getting the attention of celebrities like JK Rowling and Elon Musk, Adam explains that his method came down to “reacting to something and jumping on a story”, but also added “You can have all the social media skills and you can understand your institution, but if you don’t have that spark then it will always fall flat. Creativity and a willingness to react is key.”

“Give up ‘the fear’ and experiment all the time. Focus on what makes your [organization] unique.”

Learn more about Adam here.

This health investor shadowed Jeff Bezos and worked for Bill Gates — here’s what he learned – CNBC

Tucked away in this article about Amazon’s quest to dominate the healthcare industry—most notably via the purchase of PillPack last year for over $750M—is a line about what Microsoft Executive Peter Neupert learned by shadowing Bezos for a few days: “I learned quickly from those days that opinions don’t matter,” Neupert said. “Data matters.

Bezos talked about A/B testing (comparing two versions of a design or project), rapid customer feedback and the importance of experimenting and failing.

Why Subjective Marketing Decisions Can Lead To A Poor Customer Experience – Forbes

In this Forbes article, author Dan Gingiss explains that simple testing is not enough to optimize your customer experience—as it may only tell a part of the story: “With advanced analytics tools like multivariate testing and experimental design, marketers can learn not just what works, but why it works, and which elements of a marketing campaign are the most important.”

Microsoft Advertising rolling out experiments A/B testing tool globally – SearchEngineLand

Similar to drafts and experiments in Google Ads, experiments in Microsoft Advertising allows you to set up a duplicate of your campaign and run a test on a segment of its traffic. “This way, you can run a true A/B test within a campaign to determine whether a particular update will work well for you and your business,” writes Subha Hari, senior program manager and Piyush Naik, principal program manager, for Microsoft Advertising in the announcement.

5 Things We Learned from Analyzing 28,304 Experiments – Business2Community

Convert.com, an a/b testing tool company, looked at over 28,000 experiments and found a few interesting insights. It was intriguing to see how in-house teams compare with agencies when it comes to winning results and overall uplift.

For example, one statistic shows that survey respondents reported a 30% win-rate, while agencies found clear winners in roughly 39% of their experiments. When it came to lift, “1 out of every 7.63 experiments (13.1%) achieved a statistically significant conversion rate lift of at least 10%” while 15.84% of agency-led experiments were significant with a lift of at least 10%.

How A/B Testing at LinkedIn, Wealthfront and eBay Made Me a Better Manager – First Round

In this interview, Instacart VP of Product Elliot Shmukler explains why he uses A/B testing as a management framework, illustrating how it works to not only accelerate decisions, but also to empower the teams making them. Shmulker outlines the framework’s benefits and challenges, as well as how to implement and scale it at a startup. Any product or growth leader will learn from his data-driven approach to product and team management.

experimentation news roundup - upcoming events

Upcoming experimentation events and webinars

ConvEx 2019 – Aug 5-9 (Free Online Event)

ConvEx is a free online summit for digital marketers, conversion optimization professionals, and user experience designers across the industries aimed at encouraging experiment-driven marketing and building amazing digital experiences. Some speakers include:

  • Ronny Kohavi – Technical Fellow & Corporate Vice President, Analysis & Experimentation, Microsoft
  • Lukas Vermeer – Director of Experimentation, Booking.com
  • Rui Matos – Global Head of Conversion Optimization, Kaspersky
  • Experimentation thought leader and WiderFunnel founder, Chris Goward

How to Optimize Your E-Commerce Website at Every Customer Touch Point – Aug 15th – Online

Make sure to save your spot for our next free webinar on August 15th. We team up with intelligence platform, FullStory, to share the frameworks and success factors that fuel leading e-commerce experimentation programs. Featuring James Flory, Senior Experimentation Strategist at WiderFunnel.

Opticon19 North America – San Francisco – Sept 11-13

One of the largest experimentation events in North America. If you are in the experimentation space in the United States or Canada, you will want to attend Opticon19 in San Francisco this fall.

Opticon Europe 2019 – London – Oct 1st

Optimizely’s European conference, hosted out of London and geared towards Europe-based companies.

Current job postings in the experimentation space

Stay tuned for our next experimentation news round-up in two weeks!

some image

Discover how your experimentation program stacks up!

Benchmark your experimentation maturity with our new 7-minute maturity assessment and get proven strategies to develop an insight-driving growth machine.

Get Started