< Back to main blog page

How to get the most out of your A/B testing tools

Anecdotes on letting grandparents drive an F1 car and other tech mistakes.

By: Michael St Laurent Last updated: July 20th, 2021 10 min. read

  • A/B Testing

You wouldn’t buy an extremely expensive Formula One car and let your grandparents take it out for a Sunday drive.

Most people realize that F1 cars are incredibly sophisticated and extravagant pieces of machinery that can only be operated by highly trained race car drivers.
It’s no different with the often costly technology that is used for experimentation. It should be operated by someone with sufficient expertise who can use the tool effectively.
So why are so many marketers convinced they have the expertise to use CRO tools appropriately without training?

I was once at a conference where a sales rep for an expensive technology (>$100,000/year) said that when customers spend large sums on a tool they expect an easier to use solution. The reality is that the more the technology costs, the more features and capabilities the technology has, which actually requires a far higher level of expertise to “drive”.
Spending more on your tech is not always the solution.

You need a strategy and the expertise to be able to use the technology. Otherwise, you’re not getting the full value of your tech spend. After all, you can’t conduct a proper A/B testing program if you can’t get from A to B.

A common problem is that marketers might be attracted to buying a top of the line CRO technology tool that they feel they can grow into. However, the resource using the system may not be skilled enough to use the platform. To use a car analogy again, it’s like buying a Ferrari and driving it like a Honda Civic.

High-priced CRO Promise

Often, companies that buy advanced technology will come to the unfortunate conclusion that it won’t work for them. But in many cases the reality is that they did not allocate enough expertise to its operation. It’s a common mistake.

Marketers will want to be acquainted with the functionality they need so they can focus on what will serve them in a reasonable growth period. I have seen some companies getting sold on bells and whistles, but most of it will go unused.

As the Gartner Hype Cycle predicts, exciting technologies inevitably and quickly encourage over-inflated expectations, which then reliably crash into a trough of disillusionment. But the good news is that that experience primes people to develop more sustainable growth by using the technology appropriately.


The Gartner Hype Cycle (Source)

We have used this example in our Widerfunnel presentations because we find this is a fairly typical situation in which companies get excited and are sold on great technology. It makes sense that companies are fueled by the possibilities of what they can execute, however, there is often a lack of discussion about skillset and who the identified resources will be using the product.

Most practitioners know that CRO tools will not solve all of your problems. But technology alone is not a godsend. Technology doesn’t solve problems; it just enables solutions.

Don’t get me wrong. The CRO tools that companies buy are great – Ferrari-level great – but unless a Lewis Hamilton works for them, they won’t know how to use the technology properly. Many marketers are convinced they have the expertise to use the technology optimally, unaware that the required expertise is much more substantial than they think. We can help companies use the technology and maximize their investment.

Our goal is to fast forward clients through the innovation process, so that they can start closer to the Slope of Enlightenment stage and never have to face the host of unrealistic expectations that lead to the Trough of Disillusionment.

Unfortunately, many of our clients only come to us after they hit that painful Trough of Disillusionment stage with their new technology.

They’ve never tried testing before but hear about this amazing CRO tool. They buy it with great expectations, only to discover it is super-hard to use. They end up wasting their time achieving poor results and diminishing trust within their organizations. These clients reach the conclusion they don’t know what they’re doing with the technology but are locked into a contract for it. These clients quickly realize they need help from a service partner, and pronto.

A Techie Spending Spree

Another frequent mistake companies make is overspending on tech. Many companies spend nearly their entire budgets on technology, leaving little room for the experts that can use it to its potential. Don’t spend 60k on a testing tool while allocating 20k worth of side of the desk resources to its operations. I would love to see more companies reverse this. Find an inexpensive tool, and use that budget to commit more resources to its usage. The companies we see having the most success are having it not because they have the best technology, but because they put the appropriate resources behind their efforts.

In addition, many marketers don’t really have an experimentation strategy that goes beyond A/B testing. Strategic initiatives are not fully considered and experimentation is approached on an ad hoc basis.

People frequently underestimate the value that testing can bring to bottom lines.

While many companies devote teams for SEO or paid search work, the same does not hold true for A/B testing or conversion rate optimization. For whatever reason, companies may think such testing is worth only a few bumps in percentage points for conversion rates. In fact, the ROI is arguably double or triple what can be achieved with SEO or paid search.
If this feels like a familiar situation, it might be a sign that you have hit the wall with your current tool and need to optimize your investment by finding a CRO agency that can guide you through the next steps. An effective partner will go beyond optimizing conversion rates and use scientific methods that enable them to answer questions with high degrees of confidence. Look for a partner that offers an external perspective, one that goes beyond building tests.

A seasoned agency will be able to communicate effective strategies that have worked at various companies, across different brands, verticals and industries—giving your team an advanced approach to your CRO and experimentation program. To complete a fulsome assessment, you may want to inquire about user experience research and behavioral science to ensure you are able to gauge their full suite of services.

CRO and experimentation are not just about making a landing page look better. It’s about addressing what resonates for customers. Far from being a commodity, testing can be used as a way to learn how to validate choices and ensure you’re making the right ones that will grow rather than sink the company.

An important factor that isn’t often discussed at the outset of an engagement is how to frame messaging about digital transformation and personalization to C-suite executives and explain to them why they should funnel more of their money into testing.

We can highlight how testing and experimentation are ways to grow as a business and provide concrete examples of ROI. Some of the biggest companies in the world have grown via that philosophy.

Also, showing how testing results compare to ad spend will often justify a greater budget allocation on experimentation services. To run an effective testing program, you will require more bench strength than simply tasking an inexperienced, mid-level employee to execute .


Calculate your CRO return on ad spend with our calculator

get the calculatorarrow image


When you work with a qualified CRO agency, instead of hiring one person, you will have access to the expertise of a designer, developer, strategist, analyst and data scientist. That’s a great alternative if you have a small team and can’t afford to hire the necessary resources. For many companies, investing in ongoing services with an experienced team may make more financial sense overall and provide added benefit in the way of expertise.

Most importantly, to focus on building a sustainable and productive program, you will want to invest time and resources into driving a cultural movement. This will ensure that you not only learn how to advance the maturity of your experimentation program, but you continually educate your organization about its value to your bottom line.

Tips for optimizing your tool usage

Below are 3 tips to ensure you are getting the most out of your A/B testing tool.

1. Hire an ‘expert driver’

Too often organizations are dramatically underutilizing their A/B testing tool because they have over-bought on features, and under-bought on talent to drive them. This leads to unnecessary executive scrutiny, and the benefit of testing is overshadowed by excessive spend. Finding an expert driver is challenging, so if you cannot find one in your local market, consider bringing in an agency partner to show your team the ropes.

2. Document your Simple vs. Advanced Use Cases

Often tool upgrades are driven by fringe use-scenarios. On occasion you will run into scenarios where the technology may be insufficient but I consistently see companies pay double or triple their current technology rate in order to go from 95-99% use-case coverage. After launching 1,000s of experiments I can safely say that most businesses (barring specific technical setups) can run ~90% of tests through a simple client-side A/B testing solution. Even if you remove a technical barrier with a tool upgrade, ask yourself whether you would generate more revenue from enabling that use-case or by allocating funds to an additional developer to increase your testing output all year? I would argue the latter is a better use of money.

3. Prepare for the trough of disillusionment

Managing expectations inside of the organization is one of the most critical jobs of a testing leader. The more substantial the technology spend, the more substantial the scrutiny will be on the program (The depth of the trough is amplified by the size of the spend). The best way to manage expectations is by creating proof of value before going for a large tech spend. Get your program off the ground with an inexpensive or free solution to produce that initial value before buying the Ferrari.

Subscribe to get experimentation insights straight to your inbox.