A/B testing during the holidays: Risks and rewards

Reading Time: 6 minutes

Are you considering running A/B tests this holiday season?

Testing during the holidays may seem like a good idea — after all, a seasonal spike in traffic can help complete a test quickly — but it is important to understand how these results might be skewed.

'Hmm... Only 11 days till I have to start Christmas shopping.'
When you wait til the last minute, delivery time trumps price.

Your consumers behave differently during the holidays. They may have a higher sense of internal urgency, or they may be more concerned with factors like delivery time than with savings.

In addition, if you have made changes to your advertising spend may, you may be bringing in an audience that isn’t your usual one. If your spend is higher during the holidays, you will most likely see a higher percentage of visitors to your site coming from paid search and other marketing campaigns.

Because you are dealing with a unique consumer mindset and possibly consumers that are not within your ‘normal’ target audience, any insights you gain during holiday testing must be taken with a grain of salt. What works during a period of high urgency may not work during the rest of the year.

But testing during the holiday season isn’t necessarily a bad idea, either. ‘The holidays’ is simply one on a long list of external factors that can change your buyers’ moods and behaviors.

Whether you decide test or not over the next couple months, you can still gain valuable insights from the unique visitor behavior that comes with the holiday season.

Know before you test.

Some companies are hesitant to test during the holiday season because of the perceived financial risk of sending costly traffic to variations that might underperform, or the operational risk of introducing tests during a seasonal code freeze when little to no change is allowed by website development teams. (Sound familiar?)

But, let’s take a closer look. Refusing to test during the holidays means you will miss out on a prime time for quick test completion, which means you may actually be losing more money than you would by running a test.

For example, how do you know that your current page is performing as well as it could be? If you are not testing, you will never be able to understand what drives your customers during the holiday season.

A possible loss in the short run may generate bigger wins in the long run, by revealing important insights about how your consumers think and behave at during periods of high urgency (like the holidays).

As far as operational risk goes, many platforms, including Optimizely, have a visual editor that allows you to create your own variations without touching your site’s code―as long as the changes are simple and don’t change the functionality of the page.

For example, you can remove, resize, add, and rearrange content without changing the code. A simple test, such as changing a headline, is easy to do and doesn’t require your development team to drop what they are doing in order to code a short test.

Here are three options for how you can approach A/B testing this season.

Option 1: Keep testing and gain urgency insights

You can use the holiday season as your once-a-year opportunity to determine statistically significant winners on multiple tests in a limited timeframe. Try testing your shoppers’ sensitivity to Urgency (one of the six LIFT Model factors.)

WiderFunnel's LIFT Model.
WiderFunnel’s LIFT Model.

Consider what is important to your shoppers at different moments during the holiday season, for example:

  • Early season shoppers may be less time-sensitive than last-minute shoppers. The most effective messaging may be about “finding the right gift”.
  • For later season shoppers, the number of products left in stock and delivery timelines may be more relevant. Use stock counts, recent order counts, or offer expedited shipping options and make gift options more prominent.

One of our current clients had run A/B tests during previous holiday seasons, and because of the insights they gained, they are able to update their promotional headlines this holiday season based on past insights.

This client found that actually mentioning the name of the promotional event (Black Friday, Cyber Monday, etc.) showed a marked improvement. They are employing the following headlines at different moments this season, knowing that their consumers will respond positively because of past test results:

  • “Black Friday limited-time offer”
  • “Cyber Monday limited-time offer”
  • “Holiday limited-time offer”

Once the shopping season is over, repeat your tests to find out if the insights you gained apply to the other seasonal moments (and possibly, although unlikely, all year round). At WiderFunnel, we used this approach to increase donations over the holiday season for the International Rescue Committee (IRC), a non-profit organization.

In IRC’s case, we discovered that when external urgency was added during a disaster donation season, a different page performed better than during a non-urgency period. The organization learned some important principles about the different types of messaging to use during high and low urgency seasons from the landing page tests we conducted.

For more details about how the test was run, watch this landing page optimization webinar.

The Motivation Factor

Below is a visual representation of the Fogg Behavior Model, one of today’s most popular models for understanding persuasive design and user behavior. The FBM has three parts: motivation, ability, and triggers, asserting that “for a target behavior to happen, a person must have sufficient motivation, sufficient ability, and an effective trigger.”

Source: What causes behavior change [Infographic]
Source: What causes behavior change [Infographic]

Keep in mind that, during the holidays, consumer motivation is high, so ‘ease’ and ‘ability’ are less important. When you are developing your holiday season A/B tests, focus on tapping into users’ core motivations (e.g. time and money) rather than trying to test usability.

Option 2: Keep testing and gain revenue

If you just want to make the most of the high-traffic holiday season to make money, and if gaining insights is a lower priority for you, you can test using the multi-arm bandit (MAB) approach. MAB is an automated testing mechanism that diverts more traffic to better performing variations.

It is best reserved for times when the potential revenue gains outweigh the potential insights to be gained, or the test has little long-term value. MAB generates fewer insights because you only discover the best performing variation. You don’t get a sense of how much better one variation performs over another or which factors contributed most to the success of that variation.

Say, for example, you are running a Black Friday promotion and you’ve got a site-wide banner. This banner is only going to be around for a few days before you switch to the next holiday. So really, you just want to make the most of the opportunity and not think about it again until next year.

A bandit algorithm applied to an A/B test of your banner will help you find the best performer during the period of the experiment, and help generate the most revenue during the testing period.

While you may not be able to infer too many insights from the experiment, you should be able to generate more revenue than had you either not tested at all or gone with a traditional, even split test.

Option 3: Hold off on testing while you conduct qualitative studies

If A/B testing is out of the question for you this holiday season, then use this time to learn about your users through different means. Focus on gathering as much qualitative information as possible.

Infinity Optimization Process (TM)
WiderFunnel’s Infinity Optimization Process™ marries information gathering (Explore) with testing (Validate).

Try interviewing your customers, sales, and support staff to understand customers’ motives and why they buy from you. Don’t underestimate the value other departments can bring in helping you better know your users.

You could…

Conduct focus group and usability tests. Focus groups will help you understand your users’ feelings and opinions about your brand, your product, and your service, while usability tests will reveal how people are using your website.

Interview your sales team. Your sales people are privy to the motivations and objections of your potential customers. Here are some potential questions you could ask:

  • “What are the top 3 questions you hear from potential customers?”
  • “What are the top 3 objections you hear from potential customers?”
  • “What do potential customers say about our competitors?”
  • “Are there any particular aspects of [X] that potential and current customers don’t understand?”
  • “What aspects of [X] do potential and current customers like most? Least?”

Interview your customer support team. Your support team will help you understand the questions, fears, and feedback they get from your users. Here is a list of questions you could ask:

  • “Are there any particular areas on our site that users use as reference when speaking to you?”
  • “What was the last support question you received?”
  • “Are there any particular aspects of [X] that people don’t understand?”
  • “What aspects of [X] do people like the most? The least?”

Ideally, you can continue to test during the holiday season, but remember to limit your test timeframe to the season itself. Don’t let a test run any longer, or it will skew your results. Remember that the insights you get are specific to holidays and should be treated as such (unless follow-up tests prove otherwise).

If you can’t do any testing during the season, use the time to keep driving for insights, remembering that any results, even negative ones, can give you a clearer picture of what your customers are looking for.

Enjoy this post? Share with your friends and colleagues:

Private: Claire Vignon Keser

Claire Vignon Keser

Director of Optimization Strategy

As Director of Optimization Strategy, Claire is wholly dedicated to the success of her clients and team.