🥳 You have completed all topics in the Handbook. Click here to claim your certificate!

3. Beyond quantitative tests

While A/B-tests and similar quantitative methods are useful in scaling an experimentation program, there are many other ways of gathering data for conversion rate optimization, too. In this topic, we'll take a look at some of the more popular ones.

Quantitative data is focused, quite literally, on quantities. Quantitative analysis is not that interested in the “what” or “why”, but rather on the “how much” or “how many”.

This can be very useful, because it converges on averages, means, and trends. Outliers are typically excluded or levelled out.

If you run an A/B-test that returns a strikingly clear, statistically significant result, you can be fairly confident in the result, whether it matches your business intuition or not.

But one thing that quantitative tests have a hard time showing is individual sentiment.

What if a handful of your customers or visitors from important segments have a strong, viscerally negative reaction to the change? Could this be enough to overcome the quantitative results?

Quantitative methods are strongest when you’re consistent with them. As such, it makes little sense to go against them if there are a few outliers that diverge from the mean. However, using other, more qualitative approaches might reveal something that the aggregated data did not.

Running surveys, using screen recorders, measuring heatmaps, and collecting feedback from individual visitors will, at the very least, give you extremely valuable input for hypothesis generation in your experimentation program. And sometimes the feedback might be strong enough to question the validity of your quantitative data altogether.

As a technical marketer, remember that data is plural. It hides the complexity of individual sentiment, and it tyrannically overrules idiosyncrasies in favor of converging towards aggregations, averages, means, and trends.

However, sometimes it’s useful to pause and to dig the individual’s voice from the mass.

In this Topic, we’ll take a look at some of the technologies for gathering qualitative data from your visitors and customers.

Screen recorders

Screen recordings can be fascinating to watch.

They collect data about the visitor’s mouse movements, scrolling, keyboard actions, clicks, and what the visitor sees in the browser or app viewport at any given time.

This is typically done with a JavaScript library that the page needs to load.

After the library is loaded, it takes a “snapshot” of the page, together with all the elements, text, and images in place. Then, it constantly collects data about the user’s mouse movements and interactions with the page, superimposing these on top of the snapshot to generate a “recording” of the user’s actions.

These recordings can then be replayed in the survey tool.

From a data collection perspective, screen recordings can be very problematic, because they can inadvertently collect extremely sensitive information. Most commercial recording tools automatically mask sensitive fields such as personal data (email addresses, full names), passwords, credit card numbers, and so forth. But it’s impossible for these tools to qualify all possible manifestations of sensitive data. Thus, much is left to the discretion of the tool user to make sure that other, potentially hazardous data is masked manually.

From a data activation viewpoint, screen recordings aren’t always very useful. After all, they record an individual user’s navigation on a site, but you have no background information about these users. If you see someone struggling with adding items to the cart, is it because there is something wrong with your store’s user interface design, or is it because this individual user just had a glitch with their mouse or their browser?

However, for analyzing potential bottlenecks in the user experience flow of your site, screen recordings can be extremely valuable. If you see multiple users struggling with the same interaction, it’s definitely food for thought when optimizing the conversion paths on the site.


Heatmaps are a more quantitative version of the approach that screen recorders take.

Instead of recording every single interaction with detail as screen recorders do, heatmaps collect data from multiple visitors and show the “hotspots” on the page or screen that visitors most interacted with.

So it’s like stacking multiple screen recordings on top of each other and then highlighting the areas that most receive mouse movement, scroll actions, clicks, key presses, and so forth.

From a data collection perspective, heatmaps are less controversial than screen recordings. The snapshot is usually generic, collected by a crawler, and sensitive fields are masked both automatically and manually.

From an activation perspective, heatmaps could help you audit whether visitors are focusing on the areas of the web page that you consider important. For example, if no one is interacting with your campaign banner, it might give you a clue that the banner has been poorly designed.

Similarly, sometimes a heatmap shows users interacting with an element that you did not expect them to interact with. That might nudge you towards redesigning the element so that you can capitalize this engagement. A classic example is an email address on the page that hasn’t been formatted as a link. Visitors might try to click it with the hopes of opening it in their email software. If your heatmap shows this, it might be prudent to consider turning the email address into a proper link.

Ready for a quick break?

Based on surveys, screen recordings, and eye-tracking, it’s obvious that you should take a break. Leave the screen for a moment (and cover that webcam!), and come back refreshed and ready to tackle the rest of this Topic!

Surveys and feedback

Surveys and feedback questionnaires are usually configured so that they pop up on the page after the user has performed a certain series of tasks.


Post-purchase surveys are very common. After making a purchase, a small survey pops up to ask whether there were any issues with the checkout process, or it might have a link to celebrate your purchase on social media.

Surveys and questionnaires can be intrusive, especially when targeted to the wrong audience.


If a visitor visits your site for the first time, it is probably not a good idea to show an NPS (net promoter score) survey to them. The survey measures customer loyalty, and first-time visitors might not have a clue about your brand, never mind what it’s like being a customer.

Similarly, asking someone to subscribe to your newsletter when they just did it in the previous setting might also grind some gears.

Figuring out the best time and moment to show a survey is tricky. It’s actually a good idea to run some experiments with different surveys, toying around with timing, targeting, and messaging to see what the best combination is.

If done right, surveys and questionnaires can be extremely valuable. They might help you identify friction points in your site that would be missed with quantitative data alone.

You, as a technical marketer, might also be too “deep” in the brand to recognize the problems that visitors who do not know your brand or your products might be experiencing.

Surveys can of course give you positive feedback, too, although typically it’s more common to complain than to celebrate. Reviews, ratings, and NPS scores can all be used as social testimony about the quality of your services. Voice of the customer data can be used to improve your services and to market them as well.

Activation with qualitative data

While activation is already discussed above, it’s useful to review how qualitative feedback in its many different forms can be valuable in a marketing context.

Surveys, screen recordings, questionnaires, and even individual interviews can shed light on your customers’ and prospects’ values, pain points, preferences, and expectations. You can only go so far with intuition and hypotheses generated from your own ideas of what would be good for your brand.

While qualitative data can be difficult to aggregate into cohorts and interest groups, you might still see trends emerging from the feedback that allows you to design new marketing segments based on the qualitative feedback.


Just the action of responding to a survey puts the visitor on a different level from those who did not participate in the survey. The engagement indicates that the visitor wants to leave feedback, and as such is a signal that they are more invested in your brand than those that did not participate. This can be a useful marketing segment for targeted messaging. Just note that targeting willing participants only is not a good growth strategy – optimizing for the audiences that are not yet willing to engage is a far bigger opportunity than trying to win over those are already in your camp.

Qualitative data is directly applicable to idea and content generation. Whether it’s for generating hypotheses for your experiments or for updating the content and user experience of the site to placate those who left negative feedback, there’s potential value in every single response and recording.

With experience, you’ll learn to balance qualitative and quantitative measures appropriately.

It doesn’t make any sense to ignore a clear-as-day quantitative trend just because a handful of visitors gave a diametrically opposed viewpoint in a survey response.

At the same time, your quantitative approach might be founded on an incorrect premise or assumption about your customers, and listening closely what they have to say can be an excellent way of getting back on track.

All of this – quantitative data collection and qualitative research can be used to fuel what is known as conversion rate optimization. When you consider things more broadly, CRO is really just about growing with data. Yes, it’s founded on the principle of experimentation, but improving conversion rates, removing bottlenecks from the site, and listening to the voice of the customer is the purpose of all digital marketing, not just CRO.

Key takeaway #1: Screen recorders collect a “movie” of the user’s behavior

Screen recorders are designed to collect user activity on a web page to later replay the recording for clues about potential friction points in the user experience of the page. The user’s activities, such as mouse scrolling, clicks, and keypresses, are collected with granular timestamps. They are then compiled into a “movie” that can be replayed to show what the user did on the page. Due to the potential sensitive nature of these recordings, it’s important to mask or hide any sensitive information the user might type into form fields (passwords, personal data…).

Key takeaway #2: Heatmaps show the spots where the users dwell

Heatmaps are designed to show which parts of a web page or app screen the user dwells on or engages with the most. For example, click-based heatmaps would show the elements and areas of the page that received the most clicks. Scrolling-based heatmaps would show how deep users generally scrolled any given page.

Key takeaway #3: Qualitative tools to complement the quantitative ones

While much of CRO (and analytics for that matter) is based on the collection and analysis of quantitative data, there’s a lot of value in qualitative research, too. Sometimes asking users directly what they think about the brand, its products, its website, or its apps might give a lot of information that quantitative data could never uncover. However, it’s important to keep a level head when collecting qualitative data. Outliers are more difficult to spot, and it would be unwise to make significant adjustments based on a few survey results alone, especially if they contradict the quantitative trends.

Quiz: Beyond Quantitative Tests

Ready to test what you've learned? Dive into the quiz below!

1. What does the net promoter score (NPS) measure?

2. What does masking mean in the context of screen recorders?

3. What data do screen recorders typically collect? Select all that apply.

Your score is


What did you think about this topic?

Thanks for your feedback!

Unlock Premium Content

Simmer specializes in self-paced online courses for technical marketers. Take a look at our offering and enroll in one or more of our courses!

0/27 Topics completed

Solid work!

Complete all topics to claim your certificate.

Next Chapter