How to interpret scrollmaps for effective A/B testing

Reading Time: 6 minutes

Do you remember when people used to buy full music albums?

Like, the good ‘ol days of records, cassette tapes and CDs? The days before iTunes made it easy to be a one-off consumer, obsessed with the next, delicious single?

Well… your web page is your hit album. When you created it, you probably envisioned it in its entirety, as the sum of its parts. For you, each section on your page is a part of the narrative.

But, visitors often have short attention spans. They’re hungry for that hit single. Think of the space above the fold as that single. This is the area that captures or loses your visitors. For many businesses, the space above the fold is all anyone will ever see. (Much like Carly Rae Jepsen’s “Call Me Maybe” — did anyone download that whole album??)

call me maybe gif

That said, it’s entirely possible that your target audience still prefers complete albums; these visitors want to examine all or most of your webpage. They are patient, they’re scrollers, they want the whole story.

As a modern marketer, you know that the more you understand about your visitors, the better. Fortunately, technologies exist nowadays that allow you to capture certain aspects of visitor behavior. Scrollmaps and other tools help you visualize the visitor experience: how far down your page are visitors going, what are they clicking, and how are they consuming information?

If you’re doing or considering conversion optimization (and you should be), it’s a good idea to incorporate scrollmaps, clickmaps, and heatmaps into your analytics arsenal.

Scrollmaps are particularly useful because where you’re testing is just as important as what you’re testing. (I’ll get into this more in a second).

If you haven’t explored this aspect of your visitors’ behavior yet, don’t worry. This post is all about how to leverage the tools that are out there so that you’re not only testing the right elements, but you’re testing them in the right place.

What is a scrollmap?

Scrollmaps, heatmaps, and clickmaps are all tools that allow you to visualize visitor behavior on your webpage. Scrollmaps display a visual representation of the amount of time a visitor spent viewing each section of your page.

Scrollmaps speak with colors. Depending on the tool, red, white and yellow generally indicate areas that visitors spent the most time viewing, while teal, dark blue and black indicate areas that visitors almost never see.

How do these tools work?

Some of the most popular scrollmap tools today are Hotjar, Crazy Egg, Decibel Insight, and Clicktale. Each is a little different. Generally, you’ll tell your tool what page and for how many visitors you want it to collect data for.

A big factor here is how much traffic you have on the page. Rather than time, I would look at the sample size included e.g. pageviews or visitors. For low traffic sites you will see [user behavior] patterns emerge with just 2,000 pageviews. On higher traffic sites you might want to bump that up to 10,000.

– David Darmanin, Founder & CEO, Hotjar

Your tool will collect user data until it has sufficient pageviews, then it will generate your scroll/click/heat-map for you.

Take a look at this scrollmap from Telestream, a screencasting and video editing software:

Telestream scrollmap
Revealing scrollmap data from Telestream.

There are a few things we can gather from this scrollmap:

  • Few users are scrolling past the four image tiles, missing the important product features beneath them
  • The data seems to indicate that these 4 images are creating a “false bottom” on this page

These insights can become hugely important when you’re testing. For Telestream, we were able to use this information to form a testable hypothesis (and a winning variation).

At this point, you might be thinking, Wait a sec, do I really need to test anything? If I discover that most of my visitors don’t scroll past the fold, can’t I just move the most important information on my page above the fold?

Yes and no. Let’s say you go for it. You move the information you deem most important above the fold: your call-to-action, your main product value points etc. Maybe you see a spike in conversions, maybe you see a decrease in conversions…maybe nothing changes at all.

There are a few reasons this is problematic:

  1. You’re guessing: How do you know that the information you think is most important will be the most influential for your visitors?
  2. You’ve taken an action, but you haven’t set yourself up to look at cause and effect: Did your changes actually have any effect at all or are external factors at play?
  3. You’ve cornered yourself: You have an outcome, but you’re missing the ‘why’ >> without the ‘why’ it’s tough to move forward.

Without a control, you can’t be sure that your changes had any effect on the outcome. In fact, the only thing that you can be sure of is that you have no way of knowing whether your changes were the cause of an increase or reduction in conversions. You’re left in the dark.

Scrollmaps, while useful, are just a part of the puzzle. They allow you to create more informed hypotheses to test, based on observed visitor behaviors and patterns on your webpage. Testing is the only way to both have confidence in a change and gather further insights about your visitors.

So, yes. You still need to test when using scrollmaps.

Exploring with scrollmaps

At WiderFunnel, our testing program begins in Explore (phase 1 in the Infinity Optimization Process™). Explore is organic and expansive: it’s the data-collection phase. You want to be armed with as much information as possible before you begin hypothesizing.

infinity optimization process
The Infinity Optimization Process is iterative and leads to continuous growth and insights.

A portion of this data-collection revolves around digging into your digital analytics and user research. This is where scrollmaps, heatmaps and clickmaps join the party.

As you can see, these tools provide just a fraction of the data you want to get your hands on before you start testing. But, that doesn’t diminish their importance. For some businesses, scrollmap data can be a total game-changer.

Don’t kill your A/B test before it begins

Let’s say you are not using any extra analytics tools like scrollmaps. You’ve decided to run a test with a single variation. In this variation, you add a new section halfway down your webpage. The test runs…and runs…and runs. The data comes back showing that the variation didn’t make any difference. Naturally, you scrap this test strategy.

Here’s the main rub: when you’re A/B testing, you must capture enough conversions to reach statistical significance. I won’t get down and dirty with statistics in this post (that’s a whole other blog post series). In laymen’s terms, if a variation beats a control with 95% statistical significance, you can assume that 19 times out of 20, the result you’re seeing is not due to chance.

Suffice it to say, you should shoot for 95% statistical significance on every test you run.

If you’re testing on an area of your page that most visitors aren’t seeing, you’ll have a much harder time detecting a change in conversion rate lift. The majority of your visitors won’t see any difference between your control and your variation. If 90% of visitors aren’t scrolling past the fold, and that is where your change is, then 90% of visitors are seeing control vs. control.

The small number of people that do see your test won’t have much impact on the conversion rate because they’re such a small portion of your audience. You would need to run your test for a much longer time to reach 95% statistical significance because the conversion rate lift would be very small, requiring more traffic to detect a change.

We recently reviewed a page for a current WiderFunnel client…

…and saw that just 25% of their traffic scrolled past the fold. As a result, we knew that any test built for that page had to impact the elements above the fold if we wanted to maximize impact for the business. Otherwise, few visitors would be influenced by the change.

If we had designed a test impacting an element below the fold, it would have taken much longer for us to achieve statistically significant results. Nobody (or very few) people would have actually scrolled down the page to see what had been changed. It would be impossible to know whether the lack of impact from our test was due to the change we’d made or to the fact that nobody saw it. It would be a shame if you discounted a great idea just because of poor placement!

Don’t neglect that first single

I’ve talked about the benefit of scrollmaps in better understanding visitor behavior on your webpage. But, I haven’t yet discussed what might be motivating or demotivating your visitors to scroll in the first place.

Remember the music album analogy? It’s entirely possible that your visitors do want to listen to an entire album, but your first single (the space above the fold) turned them off.

Above the fold is where you make a first impression with visitors: good, bad or ugly. If it’s a bad impression, new visitors won’t be motivated to scroll further and will most likely bounce. If you’re just getting started with conversion optimization, it’s a good idea to test above the fold: perfect that opening message and see if they start scrolling!

What do you think about scrollmaps and similar analytics tools? How do you employ these tools in your optimization efforts? What are your favorite tools? Let us know in the comments!

Enjoy this post? Share with your friends and colleagues:

  • Fauveaux Mathieu

    be careful, all the tools interpret differently the scrollmap. For example, Hotjar does not record the user attention but only the number of user per scroll level. It’s difficult to interpret and impossible to know who is important for user.

    • Hi Faveaux,
      You’re right! Each scrollmap tool is slightly different and it’s important to get really familiar with the capabilities of whatever tool you’re using. You need to know exactly what is being tracked. That said, these tools are most useful in combination with myriad other analytics tools that help you form hypotheses for testing. If the number of users scrolling to the bottom of your webpage is high, that gives you hints about your user behavior that you can then translate into a hypothesis for your next A/B test. As I’ve mentioned in the article, scrollmaps can also help you decide where to test (fewer eyes on a change means a longer testing timeframe to meet stat significance etc.) Thanks for your comment!

  • Natasha, excellent article and points. So many users neglect the importance of testing and having the right tools for the job. Testing is such a critical component to online business and growth. Thanks for the insights and a great analysis on these scenarios.