A/B TESTING PRIORITIZATION
Solve the problem of experimentation backlog and ineffective testing prioritization
We ensure you’re testing the right ideas in the right order to yield the best returns for your organization.
Testing Program Constraints
When companies name the greatest limiting factor of their testing program, the answer is almost always unanimous – lack of engineering resources.
When examined more closely, the perceived lack of resources is usually a symptom of a greater problem; ineffective prioritization. The weight of a nearly infinite test backlog is felt by development teams when new ideas can’t be brought to life.
Our process: objectivity & prioritization
The principal factor that primarily impacts the yield of any testing program must be considered before any developer ever writes a line of code – opportunity cost: the potential loss and missed opportunity resulting from choosing one test and forgoing another.
So how can we make sure we are prioritizing our tests correctly?
There are two key considerations when prioritizing hypotheses:
- Where should I run the most impactful experiments?
- What tests should I prioritize over others?
At Widerfunnel we have developed two expert frameworks to help you maintain objectivity when prioritizing A/B tests and larger experiments.
Step 1: Where? – The PIE Framework
The first step in our approach to experimentation is determining where to test. We use a process called The PIE (Potential, Importance, and Ease) Framework to prioritize which pages, templates, & funnels to focus your testing efforts on. PIE involves using a 3-factor system to identify and queue your most important experiments first.
PIE Framework:
- Potential – Can improvements be made on this page?
- Importance – How valuable is the traffic to this page?
- Ease – How difficult is it to run an experiment on this page or template?
We help prioritize each of your potential experiments using this framework. Each zone is scored to determine an objective number ranking for each hypothesis.
Step 2: What? – The EASIER Method
Once high value areas of customer facing properties have been determined, the next step is to determine what to test. Clients often have a lot of ideas, but not all of them are feasible or meaningful right now. The EASIER Framework allows us to remove all biases from the decision making process.
EASIER Framework:
- Evidence – Quality and quantity of evidence collected
- Alignement – How well the idea aligns with business objectives
- Speed – How many days will the test need to run
- Impact – Estimated financial Impact
- Effort – Time & resources
- Reality – Likelihood of being productionalized
The EASIER scoring system not only helps determine the best hypotheses to prioritize, it also helps to reduce fallout around idea ownership. Having an agreed upon system to handle tests objectively takes politics out of the equation, and allows you to be smarter with your testing.
Make more confident business decisions.
Talk to our team of CRO experts.
Prioritization FAQs
There really is no such thing as a good or bad score with the PIE Framework. PIE is not a benchmarking tool. In order to use PIE correctly, pages need to be scored relative to each other. PIE should be used to determine which of your testable areas should be the focus of your efforts, not to compare to competitors. In order to keep scoring consistent, it is valuable to centralize scoring with one team or person with a high level of expertise and oversight.
Digital analytics is a great starting place. Look for funnel drop-offs at different stages to determine where your funnel is leaking potential customers. Consider the exit rates of pages. Are more prospects leaving the funnel than there should be on certain steps?
Conduct UXR (user research). What do customers think of your page? Have you tried using surveys? Have you asked your customer service team where people tend to get stuck? Have you watched user session recordings? When companies get stuck on potential it is usually because they don’t really have a good understanding of how customers are using their pages. Start here.
Advanced tip: Consider testing elasticity in your potential scoring. Record the avg. delta of every variation that you run in a particular zone. Over time you will uncover which pages are the most elastic. In other words, you will uncover where you have the most potential to affect a user’s propensity to purchase.
Sites are made up of different experience types. Single pages (Like a homepage or cart), templates (like a product details page or article pages), and funnels (like a checkout or lead gen wizard). You should be scoring all three types here. Most companies will include their top 8-15 experiences in the PIE ranking. Websites tend to follow the 80/20 principle, where 20% of the pages generate 80% of the impact. If you score more pages you will be spending time on scoring experiences that are not worth your effort. 90% of your tests will likely be running on the same 5 experiences so no need to waste time scoring the long-tail.
This part of PIE is almost entirely done using your analytics platform. Assuming you are using Google or Adobe Analytics this is done using custom segments. The most important aspect of building segments is to effectively match how your testing too, like Optimizely,l will ultimately calculate users. We want to align these so that your traffic and conversion rate metrics will align as closely as possible between the two platforms. This will allow you to estimate runtimes of your experiments better and prevent confusion.
For example, most testing tools default to “users” for measurement, while most analytics tools default to “sessions”. This difference is often the source of confusion in those less familiar with testing.
When creating your custom segments we recommend toggling to “users” so that it matches the testing tool. You will notice your conversion rate for users will be higher than the conversion rate you would normally report on when using sessions. This isn’t a rule, as there are certain businesses where it makes more sense to use sessions, but generally switching to users when making custom segments will help you.