462: Using qualitative data to drive product management prioritizations – with Daniel Erickson

How product managers can use AI to get more actionable insights from qualitative data

Today we are talking about using qualitative data to drive our work in product and consequently improve sales.

Joining us is Daniel Erickson, the Founder and CEO of  Viable, an AI analytics tool that enables businesses to instantly access and act on valuable insights from customer feedback, saving them hundreds of hours spent analyzing feedback. Before founding Viable, he held senior leadership roles in engineering, technology, and product.

Summary of some concepts discussed for product managers

[2:25] What is the qualitative data you have found useful for making product management decisions?

When most people think about using qualitative data in product management, they think of surveys, user interviews, or getting reactions to a prototype. There’s a huge wealth of other qualitative data that often gets ignored by product teams because it is so hard to use—for example, customer support tickets, sales call transcripts, social media mentions, interview transcripts, and product reviews. Often somebody on the team is responsible for reading through all that stuff, synthesizing it into insights, and disseminating those insights across the team. This is a very manual process, so few teams decide to do the work.

[4:22] What does that manual process typically look like?

It starts with someone on the product team who says, “We need to know more about what our customers need from us.” Then the product leader goes to some poor associate PdM and asks them to collate all of the data together. This person goes to customer support and asks for raw data or asks what the customers are saying. The customer support person gives a response off the top of their head, which is biased and is not the big picture. Then the PdM or a person on the CX team reads through the data, puts it into Excel, and adds a column for bucketing to tag the data, e.g., “checkout” or “onboarding.” The PdM may do this for several data sets, such as NPS and sales calls. They tag each piece of data, find the biggest issue, synthesize the data, and write a paragraph about the issue. That report goes to the top-level leadership.

This process takes a phenomenal amount of time, from 10-20 hours per week. You get analysis for only 5-20 buckets, and because those buckets are so broad, it’s hard to take action on that one-paragraph summary. Unless you spend hours going through every single data point, you’ll miss some nuance. It’s hard to get the fidelity of information you need to act on it.

We found that artificial intelligence is starting to help companies make better product management decisions. Computers can go through the data in less time and in a more nuanced way.

[12:53] How can we use AI for better qualitative data analysis?

The first text analytics softwares could understand what is in a word cloud and identify parts of speech, but a word cloud doesn’t give you much other than some topics you might want to pay attention to. Sentiment charts also don’t show you how to take action. Over time, we have gotten more sophisticated tools to identify different topics. Now, transformer models allow computers to understand language itself. They’re no longer breaking apart parts of speech. They’re using statistics to predict what was meant. These tools are better at detecting sarcasm and agglomerating different wording about the same topic, for example it could group together “checkout” and “cart.” These tools are much more helpful in analyzing large amounts of text.

Our AI analytics tool Viable provides analysis itself. Instead of just grouping things together and identifying themes, it analyzes these themes in the same way a qualitative analyst would. You interact with it by piping data in and asking questions.