Should You Prioritize Quantitative or Qualitative Data?

User research plays a vital role in the product design process. The more we learn about our users’ needs and wants, the better design we can create for them. And the data we collect plays a crucial part in this process.

We can define two types of data — quantitative and qualitative — and every kind of data has its own attendant type of research. On the surface, quantitative and qualitative research can look similar. After all, they both analyze how users interact with a product. They serve different purposes, however, and you need to understand the distinctions. Once you grasp the differences between quantitative and qualitative data, you can understand when and how best to use each of them.

Guidelines for Collecting Data

No matter what type of data you want to collect, you should always keep the following four things in mind:

1. Have a clear goal for your research. Why do you want to collect data in the first place? A clear goal will make your research more focused. For example, you might want to optimize the product checkout experience to reduce the total number of cart abandonments by 50 percent. In this case, you will focus only on checkout user flow and collect key conversion metrics targeted at solving this problem.
    
2. What do you expect to achieve? What is the product’s expected performance? You need to have a reference point to interpret your data. A reference point might be an expected task-completion time, success rate or some other measurable metric. In our hypothetical situation, you might want the average checkout experience to be less than a minute.
    
3. Number of test participants. How many people should participate in your research? Although every product is different and suggesting a one-size-fits-all number of test participants is impossible, I can offer a very good starting point. For quantitative research, you need a statistically significant number of people to make a data-informed decision. Quantitative research typically requires dozens of test participants (typically, more than 20 to get meaningful results). Qualitative research, on the other hand, involves a small number of users. According to the NNGroup, it’s possible to uncover 85 percent of potential usability problems with just five users.
    
4. Relevant demographics. It doesn’t make much sense to collect data from people who won’t use your product. You need to recruit participants who match your target demographics as defined in your user persona.

Quantitative Data

Quantitative data can be measured in numbers. This type of data answers “How?” questions. For example:

• How many people visited this page in the last month?
   
• How much time does it take for a user to create an account in our service?

Qualitative data is typically provided in the form of metrics (i.e., conversion rate, average time on task, task competition rate, etc.).

When to Do Quantitative Research

The goal of quantitative research is to see how users interact with a product and identify areas where they might need extra help. Quantitative research can be done at any stage of the product design process, but it’s the most effective at the validation stage, either during usability testing or after the product’s release to the market. It helps researchers to gather data about what users do with a product and find patterns in their interactions. This information helps the product team to create a foundation for further benchmarking.

Quantitative research is also very helpful for calculating return on investment. You can employ quantitative methods during redesign iterations to compare the new version’s performance with a previous one. These findings are beneficial when you want to convince executives that your design team is moving in the right direction.

Quantitative Research Methods

Here are a few popular quantitative research methods:

• Surveys. A rating survey (“Please rate your experience on a scale from one to five”) and questionnaires with set choices (“Which feature from this list is the most valuable for you?”) are by far the most commonly used quantitative research methods.
   
• A/B and multivariate testing. A/B and multivariate testing can help you compare different versions of your design and see which one performs better. A/B testing involves creating two different versions of the same user interface element (for instance, different colors for a call to action button on a landing page) and then shows each version to different users to see which version performs best. Multivariate testing is based on the same idea but involves testing a few UI elements at once (i.e., modifying the color, copy text and position of a call to action button).
   
• Web analytics tools. Tools like Google Analytics and Hotjar can help you collect information about what people do in your live product — where they go, what they click and what features they use.
   
• Eyetracking studies. Eyetracking is a technique that tracks users’ eyes as they move across an interface. When many participants perform the same task on the same interface, researchers can easily notice any trends.

Practical Tips for Quantitative Research

The goal of quantitative research is to reduce the risk of biased results, which give you incorrect assumptions about user behavior. They can lead the team in the wrong direction and make them invest time and effort in building something that does not bring any value to the users. Biased results can be come from the participants’ side (typically caused by social desirability effect or incorrectly understood assignments) or the researchers’ side (when researchers interpret the data to prove their point of view). Here are a few simple tips that will help you minimize bias:

• Do not vary the study conditions between sessions. Ensure that your study sessions are all run in the same environment. Do not modify the tasks that you want participants to complete along the way. For example, do not make one session in person and another remote because it will be harder to analyze and compare the results.
   
• Make the assignments clear. All participants should understand the same thing when they read the task. The instructions should be crystal clear so participants know exactly what they should do. Don’t leave any room for interpretation or you could see too much variance in the results.
   
• Invite multiple researchers to analyze data. Collect the feedback from a few researchers to see how they interpret the data.

Qualitative Data

Whereas quantitative research is focused on finding patterns, qualitative research seeks to discover the underlying meanings of those patterns. Qualitative data is generally non-numerical information that helps researchers gain deep contextual understandings of users and explain their behavior. Researchers aim to find answers to “Why?” questions like:

• Why did people visit this particular page?
   
• Why are people more interested in this feature rather than other features?

When to Do Qualitative Research

Qualitative research is typically conducted early on in projects because the insights it reveals can dramatically alter product design. It can directly inform the design process. By identifying the core product design problems right at the beginning of the design process, it’s possible to cut both the cost and time of production.

Qualitative research can also be valuable as a follow-up activity for quantitative research. For example, when you know, based on quantitative data, that 60 percent of your users can’t complete a particular task in your product, you probably want to conduct qualitative research to better understand what problems are derailing the users.

Qualitative Research Methods

Researchers typically obtain qualitative data through first-hand observation. Any of the following methods are common:

• In-depth user interviews. This is a process in which a researcher asks one participant questions about a topic of interest (i.e., a particular part of a product) to learn about it from the user’s perspective. For qualitative research, a researcher should ask open-ended questions to generate more behavioral data. For instance, you should ask more questions like “How do you feel about this product?” to give the user as much leeway as possible in their response.
   
• Focus groups. A focus group is a small but demographically similar group of people whose reactions to a product are studied by researchers. The group typically consists of six to eight participants. They meet and discuss a product for about two hours. This discussion is moderated by a researcher who asks relevant questions and also maintains the group’s focus.
   
• Contextual inquiry. Contextual inquiry involves observation of users in a real-world working environment as they interact with a product or service. The information produced by contextual inquiry is highly reliable since researchers see how people interact with a product or service in their natural environment. It’s also highly detailed, meaning that this type of research helps researchers better understand the context of a user’s interaction with a product.

Practical Tips for Qualitative Research

In comparison with its quantitative counterpart, qualitative research offers much more freedom both for researchers and participants. Here are a few things you should consider when running research:

• Use a think-aloud protocol for usability testing. This protocol allows participants to talk about their experiences, thoughts and feelings as they interact with a product so that researchers can gather more valuable insights about real user behavior.
   
• Strive for flexibility of research. If you discover that a specific task doesn’t give you the insights you need, rework it before running the next session with a new participant.
   
• Record videos of user interactions. The videos from qualitative research can be very useful for further analysis. It’s also possible to use videos to convince the team members and stakeholders to invest time in improving user experience since they will see for themselves the problem that participants experience. 

More on the Ins and Outs of ResearchMake Your Product Research Matter

Quantitative or Qualitative Data? Why Not Both?

Quantitative and qualitative data are not competitors. Rather, they go hand in hand by reinforcing each other. Both qualitative and quantitative data are essential in the product design cycle because they help product teams iterate design and measure the results of a new interaction. Qualitative data can help identify the areas where users face problems and ideate a solution to these problems. Once you introduce changes in your design and release a new version, this version can be evaluated and compared against the initial version using quantitative data.