< Back to main blog page

Purists and Pragmatists in experimentation

Words by: James Flory Last updated: December 17th, 2021 9 min. read

  • A/B Testing
  • Conversion Rate Optimization

Introduction

Every business can agree: more experimentation gives your data analysts valuable information to make smarter data-driven decisions. What not every business agrees on, however, is the form that this experimentation should take. In this article, we’ll investigate the difference between the two prevailing experimentation philosophies: purists vs. pragmatists.

Who are Purists and Pragmatists?

The clash between purists and pragmatists is an age-old question, and one that every organization has to face once it grows large enough.

  • Purists are people who focus on correctness, perfection, and following the letter of the law.
  • Pragmatists are people who focus on practicality, efficiency, and getting a working solution in place.

Both perspectives are valuable and necessary, and they have an impact on multiple downstream questions concerning your data and brand:

  • Data purists are people who want to collect as much information as possible and proceed with extreme confidence. Data pragmatists are people who want to move quickly and are comfortable with a reasonable level of risk.
  • Brand purists want to design tests that maintain an optimal user experience. Brand pragmatists want to test the waters fast with the minimum amount of effort for iteration.

The history of A/B testing

A/B testing concepts emerged into the mainstream in the mid-20th century primarily in the medical field. This helped emphasize the importance of a high degree of statistical significance. With people’s health on the line, it was widely accepted by the industry that 95-99% became the confidence levels that had to be crossed for something to be considered valid. A higher degree of confidence is generally correlated to the cost of being incorrect (like a life-threatening outcome). The results of high-risk medical trials, for example, needed a stronger level of confirmation than did the results of marketing experiments (such as whether to send a letter or a postcard).

Modern A/B testing for business, backed by technology and the scientific method, started in the 1990s. For instance, tech companies such as Google used A/B testing to find the best number of search results to display on each web page, and even the perfect shade of blue to use for their links.

Keeping an agile mindset

For every new test, ask yourself: “How important is it that we get the right answer to this question/experiment?” Do you really need to reach a high level of statistical significance, or is it only a minor change that doesn’t require a higher degree of accuracy?

There can be large opportunity cost in chasing purism, as suggested by the concept of expected value of perfect information (EVPI) in decision theory. The EVPI refers to the price (i.e., the amount of profit lost) that you would be willing to pay in order to receive perfect information about something. (See the article “The Concept of EVI: Expected Value of Information” for more details.)

Jeff Bezos, for his part, is a pragmatist on this question, according to his “70 percent rule.” He writes:

“Most decisions should probably be made with somewhere around 70 percent of the information you wish you had. If you wait for 90 percent, in most cases, you’re probably being slow.”

Data Purists vs. Data Pragmatists

The evolution of the testing industry

Initially, the testing industry believed that the accuracy of a given experiment was its most important component. Over time, however, people realized that it’s highly impractical to reach a 95% level of statistical significance for every data point.

The industry has evolved from simple button color changes to a modern approach with multiple different methodologies. Instead of always striving for perfect information, teams are more willing to adopt a practical, holistic stance.

For example, Netflix performs what it calls “quasi-experiments” for scenarios where full A/B testing is impossible. Instead of showing two groups of people two different marketing campaigns, Netflix seeks to determine the effects of running a campaign in a specific location. However, doing these “quasi-experiments” inherently involves a level of uncertainty, acknowledging that it may not be possible to remove all of the noise from the data.

This kind of approach is a clear example of data pragmatism. According to pragmatists, if you’re at point A and you want to get to point Z, it’s fine to skip over six steps in between, rather than being slowly and methodically iterative.

On the other hand, data purists usually believe that you should perform small, incremental modifications. Purists want to know the precise causes of why a given change succeeded or failed, instead of pursuing an overarching vision.

Risk mitigation with experimentation

There isn’t a right or wrong stance here. Consider the particular needs of your organization. A startup competing to be the first viable product in the market can afford to take more risks than an established Enterprise where a 0.1% difference can cost the company millions in profit.

Risk mitigation is prudent when you’re considering big-ticket changes that can have a drastic impact on how you conduct your business. When making large changes to your business model, your teams will want to wait longer for the right outcomes, since the risk is much higher in this scenario. This is exactly the right time for data purism.

However, data purists and data pragmatists both have a key role to play when “de-risking” your business ideas. Questions such as the right image or sales copy to use involve much less business risk, allowing for a pragmatist approach.

Before deciding which stance to take, ask yourself questions like:

  • What are the budget requirements of this testing project?
  • Who should be involved in the decision-making process to decide when the right level of statistical significance has been achieved?
  • What are the risk tolerance and expectations of higher-ups on this project?
Trustworthiness

A central philosophy that must remain consistent between both purists and pragmatists is the concept of trustworthiness. The pragmatist view should not be conflated with being less trustworthy. Varying degrees of accuracy are achieved through adjusting the significance level of the experiments run to meet the predetermined risk tolerance and accuracy tradeoffs of your particular business. It is still possible to run untrustworthy experiments as a data purist using bad data, just as it is still possible to run highly trustworthy experiments to lower levels of significance. An experiment can be trusted if we know and understand the balance between the acceptable risks and rewards when we make a business decision.

Brand Purists vs. Brand Pragmatists

“It’s just a test”

The question of purism vs. pragmatism spills over not just into the resulting data, but also into the design of the test itself.

Brand purists want the experiments to be perfect, solving for every possible user scenario and outcome. On the other hand, brand pragmatists want to test a concept first, and then only put effort into perfecting it if it actually works.

The disadvantage of brand purism is that it loses one of the biggest benefits of testing in the first place. If you devote too many resources to building “perfect” experiences from the start, then you’re not actually saving time and effort. This causes your organization to lose agility and get bogged down.

The painted door experiment

Suppose that you want to add some new feature to your e-commerce website—for example, using an alternate form of payment. Brand pragmatists would encourage you to first simply add another button during the checkout process, seeing how many users click on it before you actually implement this functionality.

This kind of experimentation is known as “painted door testing.” The term comes from the analogy of painting a door on a wall and seeing who tries to walk through it. Once you have proof that there’s demand for this feature, you can get started building it.

Brand purists hate this kind of “painted door test” because your website now contains dead functionality, which could harm the user experience (just like people could walk into the wall). However, there are ways around this, such as offering a temporary discount code as an apology to users for the unavailable feature.

Conclusion

In most cases and settings, a pragmatic approach will be the right fit for your iterative testing process, and can still establish statistically significant results. By not investing so much effort in any given test, you can have more resources to devote to more tests and learn more about your customers. For large-scale, business-critical changes, however, the purist approach is still invaluable.

Do you need a trusted agency partner for your testing initiatives? Widerfunnel is here to help. We are a leading conversion rate optimization (CRO) agency with the right skills and experience to help deliver powerful results for our clients. To learn how Widerfunnel can assist your testing efforts, get in touch with our team today for a chat about your business needs and objectives.

Subscribe to get experimentation insights straight to your inbox.