As a data science practitioner, think about all the data you have at your disposal. Then think about the latest data science methods you know about. Now, try answering these two questions: Why do your users adopt or reject your products? And how do they use these products?
Both may sound simple, but they’re trick questions. You can much more easily figure out which features of the products are being adopted, when and where they’re adopted, and perhaps even by whom.
Answering “why” and “how” questions with analytics, however, is much more complicated because you need to better understand your users, the contexts in which they operate, their considerations, and their motivations. Though answers to these questions are critical, data science teams cannot always rely only on assumptions, models, and numbers to understand the choices users make and the decisions that lead them to use a product in the ways they do.
The purpose of this little thought exercise is to suggest that, while analytics has many advantages, it also has its limits. Recognizing these limits will help you broaden your insight and become more innovative.
But to achieve this, you’ll need to engage in exploration where parameters and variables are less known, assumptions are mostly absent, and curiosity abounds. You’ll need to think in a way that diverges from how you were trained, and you’ll need to use data and research methods fundamentally different from those you usually rely on.
In short, think about incorporating qualitative research into your analytics process.
Qualitative Research Can Answer the ‘Why’ and ‘How’ Questions
Typical analyses involve getting data from users’ devices, their logged activities, or through user experiments such as A/B testing. But to answer “why” and “how,” you’ll need to learn the perspectives, meanings, and considerations of users from them directly.
Instead of top-down analytics, gather data by getting out into the “field” — the contexts in which your users operate. Rather than rely on known hypotheses and existing variables to carry out deductive work (by far the more common form of analysis), immerse yourself in an inductive process of qualitative research.
The data collected in qualitative research is different from what you’re used to in data science. This is how it works:
- Observations: Observations from the field are done while the researcher takes notes on users and their behaviors, which can take place in a natural setting or in a lab. Viewing a user using a product — a website, an app, a gadget — are common examples.
- Document analysis: Because observations are done to study choices and behaviors performed on the spot, to gain longitudinal data, the researcher will often rely on user-generated documents. There are two types of these documents: community records and diaries. The first refers to content that community members generate (think Reddit on a topic of interest). In the second, the researcher asks users to record their perspectives systematically over time in the form of a diary.
- Interviews: In-depth interviews are performed with informants participating in observational studies and based on insight the researcher gained from documents. Rather than closed-ended questionnaires, these are conducted with open-ended questions to elicit more information from the perspective of the participants (for example, “Why did you use the app the way you did?”).
Note that, unlike traditional data used in data science, qualitative data are multilayered and complex. Field observations are tied to notes, the notes are then connected to interviews, both are then connected to the documents.
It’s not a linear process, and by going back-and-forth between these interconnected layers of data, patterns emerge, research questions get refined, new behaviors and characteristics identified, and insights gained. And because the data is intentionally collected mostly in an unstructured fashion (meaning not answers to specific, close-ended questions) the input reflects different perspectives.
All of this can lead to refined questions and hypotheses to further pursue using data science tools.
Qualitative Work In Data Science Is Uncommon, but Shows Promising Results
In business contexts, qualitative research is mainly reserved for studies of user experience (UX), product and UX design, and innovation. This intensive research work is largely disconnected from the work done by data science teams.
However, if your data context involves people, you should consider bridging this disconnect.
Take, for instance, an engineering team at Indeed that realized they needed to create a new measure for lead quality but didn’t have enough background about leads and how to assess their properties. So they spent some time observing, interviewing, and analyzing documents from their account executives. By analyzing these data, they identified features that they’d not considered before and developed the measure they were after.
Being able to collect data on new features and designing machine learning models added significant value. But they realized that as the market, user needs, and their platform continued to evolve, it was important to return to collecting qualitative data from time to time to further inform their analytics pipeline. This ongoing integration of qualitative data and big data resulted in millions of dollars in added revenue.
Or consider the results of having qualitative research and data science teams work together at Spotify. Despite having a wealth of user data at the online streaming service, the company still needed to make sense of users’ behavior when receiving ads. The data science team followed the standard approach and performed an A/B test (with the intervention being skippable ads and the control being the standard ad experience). The results led the data science team to identify distinct behavior profiles.
Interestingly, the company also asked qualitative researchers to directly study users. Their findings were fundamentally different. For instance, they found that some of these profiles had nothing to do with inherent choices but were actually more an outcome of confusion about features and presented information.
Learning from this experience, the company started embracing a mixed methods approach (where qualitative data is integrated with more structured big data) to leverage the benefits of both approaches. They established a common research query, devised a process where researchers continuously communicated, and then “triangulated” their insight with both qualitative and quantitative data.
The result was more comprehensive research data, where business and design decisions, such as notifying users explicitly about ad skip limits, were based upon insight gained from users and data about users.
How to Start Integrating Qualitative Research Into Analytics
There are several ways for a data science team to engage with qualitative data:
- Create curiosity by challenging your analytics team. Start by generating a list of your main data features, hypotheses, and overall research questions. Then ask your team about the assumptions and contexts related to the list. This is where asking some of those “why” and “how” questions is likely to be useful.
- Educate about the process of immersive research and the types of data collected in this type of work. This book is an excellent example for general applications that involve both traditional more structured (quantitative) research, the more immersive (qualitative) research, and suggestions on how to combine them. This book teaches how to derive insight from the data from the ground up.
You should also familiarize yourself with the three data collections methods (observations, interviews, and document analysis) that are at the core of qualitative research:
- If you prefer videos, check out this TEDx talk on ethnography, this example of an interview in the healthcare context, or this overview of document analysis.
- If you prefer short reads, here’s a column from a UX researcher on the user perspective, a Shopify blog post on interviews from the perspective of design research, and a research article on a study of intensive care unit patient families using diaries.
Finally, connect with researchers who have experience with immersive research, inside and outside your company. They can help you as you are thinking about ways to collect and analyze qualitative data. And perhaps you can offer to help them with parts of the tedious analysis of the more quantitative bit of their data too.
