Quantitative data is focused, quite literally, on quantities. Quantitative analysis is not that interested in the “what” or “why”, but rather on the “how much” or “how many”.
This can be very useful, because it converges on averages, means, and trends. Outliers are typically excluded or levelled out.
If you run an A/B-test that returns a strikingly clear, statistically significant result, you can be fairly confident in the result, whether it matches your business intuition or not.
But one thing that quantitative tests have a hard time showing is individual sentiment.
What if a handful of your customers or visitors from important segmentsWhen data is grouped by property, attribute, or value, it is segmented. When building audiences for ads, for example, you need to choose for which user segments to target the ads. have a strong, viscerally negative reaction to the change? Could this be enough to overcome the quantitative results?
Quantitative methods are strongest when you’re consistent with them. As such, it makes little sense to go against them if there are a few outliers that diverge from the mean. However, using other, more qualitative approaches might reveal something that the aggregated data did not.
Running surveys, using screen recorders, measuring heatmaps, and collecting feedback from individual visitors will, at the very least, give you extremely valuable input for hypothesisAn assumption based on research that you want to prove or reject with experimentation. Minimally, it needs to include what you intend to change, what the expected outcome is, and what your rationale for the hypothesis is. generation in your experimentation program. And sometimes the feedback might be strong enough to question the validity of your quantitative data altogether.
As a technical marketer, remember that data is plural. It hides the complexity of individual sentiment, and it tyrannically overrules idiosyncrasies in favor of converging towards aggregations, averages, means, and trends.
However, sometimes it’s useful to pause and to dig the individual’s voice from the mass.
In this Topic, we’ll take a look at some of the technologies for gathering qualitative data from your visitors and customers.
Screen recorders
Screen recordings can be fascinating to watch.
They collect data about the visitor’s mouse movements, scrolling, keyboard actions, clicks, and what the visitor sees in the browser or app viewportThe area of the web page or app visible to the user at any given time. It is constrained by the size of the browser window and the resolution of the user's device. at any given time.
This is typically done with a JavaScriptJavaScript is the main language of the dynamic web. The web browser renders the HTML source file into a dynamic document that can be interacted with using JavaScript. libraryAnother word for a file that contains code which can be utilized by downloading the library into the application. For example, when the web browser loads JavaScript files from vendors, those are frequently called libraries. that the page needs to load.
After the libraryAnother word for a file that contains code which can be utilized by downloading the library into the application. For example, when the web browser loads JavaScript files from vendors, those are frequently called libraries. is loaded, it takes a “snapshot” of the page, together with all the elements, text, and images in place. Then, it constantly collects data about the user’s mouse movements and interactions with the page, superimposing these on top of the snapshot to generate a “recording” of the user’s actions.
These recordings can then be replayed in the survey tool.
From a data collection perspective, screen recordings can be very problematic, because they can inadvertently collect extremely sensitive information. Most commercial recording tools automatically mask sensitive fields such as personal data (email addresses, full names), passwords, credit card numbers, and so forth. But it’s impossible for these tools to qualify all possible manifestations of sensitive data. Thus, much is left to the discretion of the tool user to make sure that other, potentially hazardous data is masked manually.
From a data activation viewpoint, screen recordings aren’t always very useful. After all, they record an individual user’s navigation on a site, but you have no background information about these users. If you see someone struggling with adding items to the cart, is it because there is something wrong with your store’s user interface design, or is it because this individual user just had a glitch with their mouse or their browser?
However, for analyzing potential bottlenecks in the user experience flow of your site, screen recordings can be extremely valuable. If you see multiple users struggling with the same interaction, it’s definitely food for thought when optimizing the conversion paths on the site.
Heatmaps
Heatmaps are a more quantitative version of the approach that screen recorders take.
Instead of recording every single interaction with detail as screen recorders do, heatmaps collect data from multiple visitors and show the “hotspots” on the page or screen that visitors most interacted with.
So it’s like stacking multiple screen recordings on top of each other and then highlighting the areas that most receive mouse movement, scroll actions, clicks, key presses, and so forth.
From a data collection perspective, heatmaps are less controversial than screen recordings. The snapshot is usually generic, collected by a crawlerA machine that downloads and parses content on the web. It follows links to find additional content to fetch. Search engine crawlers use the parsed information to build the search engine index., and sensitive fields are masked both automatically and manually.
From an activation perspective, heatmaps could help you audit whether visitors are focusing on the areas of the web page that you consider important. For example, if no one is interacting with your campaign banner, it might give you a clue that the banner has been poorly designed.
Similarly, sometimes a heatmap shows users interacting with an element that you did not expect them to interact with. That might nudge you towards redesigning the element so that you can capitalize this engagement. A classic example is an email address on the page that hasn’t been formatted as a link. Visitors might try to click it with the hopes of opening it in their email software. If your heatmap shows this, it might be prudent to consider turning the email address into a proper link.
Ready for a quick break?
Based on surveys, screen recordings, and eye-tracking, it’s obvious that you should take a break. Leave the screen for a moment (and cover that webcam!), and come back refreshed and ready to tackle the rest of this Topic!
Surveys and feedback
Surveys and feedback questionnaires are usually configured so that they pop up on the page after the user has performed a certain series of tasks.
Example
Post-purchase surveys are very common. After making a purchase, a small survey pops up to ask whether there were any issues with the checkout process, or it might have a link to celebrate your purchase on social media.
Surveys and questionnaires can be intrusive, especially when targeted to the wrong audience.
Example
If a visitor visits your site for the first time, it is probably not a good idea to show an NPSA survey type that asks "how likely are you to recommend us to others?". The scale is 0 to 10, with 9 and 10 considered "promoters", 7 and 8 as "passives", and 0 through 6 as "detractors". (net promoter scoreA survey type that asks "how likely are you to recommend us to others?". The scale is 0 to 10, with 9 and 10 considered "promoters", 7 and 8 as "passives", and 0 through 6 as "detractors".) survey to them. The survey measures customer loyalty, and first-time visitors might not have a clue about your brand, never mind what it’s like being a customer.
Similarly, asking someone to subscribe to your newsletter when they just did it in the previous setting might also grind some gears.
Figuring out the best time and moment to show a survey is tricky. It’s actually a good idea to run some experiments with different surveys, toying around with timing, targeting, and messaging to see what the best combination is.
If done right, surveys and questionnaires can be extremely valuable. They might help you identify friction points in your site that would be missed with quantitative data alone.
You, as a technical marketer, might also be too “deep” in the brand to recognize the problems that visitors who do not know your brand or your products might be experiencing.
Surveys can of course give you positive feedback, too, although typically it’s more common to complain than to celebrate. Reviews, ratings, and NPSA survey type that asks "how likely are you to recommend us to others?". The scale is 0 to 10, with 9 and 10 considered "promoters", 7 and 8 as "passives", and 0 through 6 as "detractors". scores can all be used as social testimony about the quality of your services. Voice of the customer data can be used to improve your services and to market them as well.
Activation with qualitative data
While activation is already discussed above, it’s useful to review how qualitative feedback in its many different forms can be valuable in a marketing context.
Surveys, screen recordings, questionnaires, and even individual interviews can shed light on your customers’ and prospects’ values, pain points, preferences, and expectations. You can only go so far with intuition and hypothesesAn assumption based on research that you want to prove or reject with experimentation. Minimally, it needs to include what you intend to change, what the expected outcome is, and what your rationale for the hypothesis is. generated from your own ideas of what would be good for your brand.
While qualitative data can be difficult to aggregate into cohorts and interest groups, you might still see trends emerging from the feedback that allows you to design new marketing segmentsWhen data is grouped by property, attribute, or value, it is segmented. When building audiences for ads, for example, you need to choose for which user segments to target the ads. based on the qualitative feedback.
Example
Just the action of responding to a survey puts the visitor on a different level from those who did not participate in the survey. The engagement indicates that the visitor wants to leave feedback, and as such is a signal that they are more invested in your brand than those that did not participate. This can be a useful marketing segmentWhen data is grouped by property, attribute, or value, it is segmented. When building audiences for ads, for example, you need to choose for which user segments to target the ads. for targeted messaging. Just note that targeting willing participants only is not a good growth strategy – optimizing for the audiences that are not yet willing to engage is a far bigger opportunity than trying to win over those are already in your camp.
Qualitative data is directly applicable to idea and content generation. Whether it’s for generating hypothesesAn assumption based on research that you want to prove or reject with experimentation. Minimally, it needs to include what you intend to change, what the expected outcome is, and what your rationale for the hypothesis is. for your experiments or for updating the content and user experience of the site to placate those who left negative feedback, there’s potential value in every single response and recording.
With experience, you’ll learn to balance qualitative and quantitative measures appropriately.
It doesn’t make any sense to ignore a clear-as-day quantitative trend just because a handful of visitors gave a diametrically opposed viewpoint in a survey response.
At the same time, your quantitative approach might be founded on an incorrect premise or assumption about your customers, and listening closely what they have to say can be an excellent way of getting back on track.
All of this – quantitative data collection and qualitative research can be used to fuel what is known as conversion rate optimizationConversion rate optimization refers to running tests and experiments with the purpose of improving the likelihood of conversion for visitors to your sites and apps.. When you consider things more broadly, CROConversion rate optimization refers to running tests and experiments with the purpose of improving the likelihood of conversion for visitors to your sites and apps. is really just about growing with data. Yes, it’s founded on the principle of experimentation, but improving conversion ratesThe ratio of conversion events to visitors or sessions. A high conversion rate usually means that your visitors are performing actions that are beneficial to your business., removing bottlenecks from the site, and listening to the voice of the customer is the purpose of all digital marketing, not just CROConversion rate optimization refers to running tests and experiments with the purpose of improving the likelihood of conversion for visitors to your sites and apps..
Key takeaway #1: Screen recorders collect a “movie” of the user’s behavior
Screen recorders are designed to collect user activity on a web page to later replay the recording for clues about potential friction points in the user experience of the page. The user’s activities, such as mouse scrolling, clicks, and keypresses, are collected with granular timestamps. They are then compiled into a “movie” that can be replayed to show what the user did on the page. Due to the potential sensitive nature of these recordings, it’s important to mask or hide any sensitive information the user might type into form fields (passwords, personal data…).
Key takeaway #2: Heatmaps show the spots where the users dwell
Heatmaps are designed to show which parts of a web page or app screen the user dwells on or engages with the most. For example, click-based heatmaps would show the elements and areas of the page that received the most clicks. Scrolling-based heatmaps would show how deep users generally scrolled any given page.
Key takeaway #3: Qualitative tools to complement the quantitative ones
While much of CROConversion rate optimization refers to running tests and experiments with the purpose of improving the likelihood of conversion for visitors to your sites and apps. (and analytics for that matter) is based on the collection and analysis of quantitative data, there’s a lot of value in qualitative research, too. Sometimes asking users directly what they think about the brand, its products, its website, or its apps might give a lot of information that quantitative data could never uncover. However, it’s important to keep a level head when collecting qualitative data. Outliers are more difficult to spot, and it would be unwise to make significant adjustments based on a few survey results alone, especially if they contradict the quantitative trends.