Experiencing Data w/ Brian T. O’Neill  (UX for AI Data Products, SAAS Analytics, Data Product Management)

162 - Beyond UI: Designing User Experiences for LLM and GenAI-Based Products


Listen Later

I’m doing things a bit differently for this episode of Experiencing Data. For the first time on the show, I’m hosting a panel discussion. I’m joined by Thomson Reuters’s Simon Landry, Sumo Logic’s Greg Nudelman, and Google’s Paz Perez to chat about how we design user experiences that improve people’s lives and create business impact when we expose LLM capabilities to our users. 

 

With the rise of AI, there are a lot of opportunities for innovation, but there are also many challenges—and frankly, my feeling is that a lot of these capabilities right now are making things worse for users, not better. We’re looking at a range of topics such as the pros and cons of AI-first thinking, collaboration between UX designers and ML engineers, and the necessity of diversifying design teams when integrating AI and LLMs into b2b products. 

 

Highlights/ Skip to 

  • Thoughts on how the current state of LLMs implementations and its impact on user experience (1:51)
  •  The problems that can come with the "AI-first" design philosophy (7:58)
  •  Should a company's design resources be spent on go toward AI development? (17:20)
  • How designers can navigate "fuzzy experiences” (21:28)
  • Why you need to narrow and clearly define the problems you’re trying to solve when building LLMs products (27:35)
  • Why diversity matters in your design and research teams when building LLMs (31:56)
  •  Where you can find more from Paz, Greg, and Simon (40:43)
  •  

    Quotes from Today’s Episode
    • “ [AI] will connect the dots. It will argue pro, it will argue against, it will create evidence supporting and refuting, so it’s really up to us to kind of drive this. If we understand the capabilities, then it is an almost limitless field of possibility. And these things are taught, and it’s a fundamentally different approach to how we build user interfaces. They’re no longer completely deterministic. They’re also extremely personalized to the point where it’s ridiculous.” - Greg Nudelman (12:47)
    • “ To put an LLM into a product means that there’s a non-zero chance your user is going to have a [negative] experience and no longer be your customer. That is a giant reputational risk, and there’s also a financial cost associated with running these models. I think we need to take more of a service design lens when it comes to [designing our products with AI] and ask what is the thing somebody wants to do… not on my website, but in their lives? What brings them to my [product]? How can I imagine a different world that leverages these capabilities to help them do their job? Because what [designers] are competing against is [a customer workflow] that probably worked well enough.” - Simon Landry (15:41)
    • “ When we go general availability (GA) with a product, that traditionally means [designers] have done all the research, got everything perfect, and it’s all great, right? Today, GA is a starting gun. We don’t know [if the product is working] unless we [seek out user feedback]. A massive research method is needed. [We need qualitative research] like sitting down with the customer and watching them use the product to really understand what is happening[…] but you also need to collect data. What are they typing in? What are they getting back? Is somebody who’s typing in this type of question always having a short interaction? Let’s dig into it with rapid, iterative testing and evaluation, so that we can update our model and then move forward. Launching a product these days means the starting guns have been fired. Put the research to work to figure out the next step.” - (23:29) Greg Nudelman
    • “ I think that having diversity on your design team (i.e. gender, level of experience, etc.) is critical. We’ve already seen some terrible outcomes. Multiple examples where an LLM is crafting horrendous emails, introductions, and so on. This is exactly why UXers need to get involved [with building LLMs]. This is why diversity in UX and on your tech team that deals with AI is so valuable. Number one piece of advice: get some researchers. Number two: make sure your team is diverse.” - Greg Nudelman (32:39)
    • “ It’s extremely important to have UX talks with researchers, content designers, and data teams. It’s important to understand what a user is trying to do, the context [of their decisions], and the intention. [Designers] need to help [the data team] understand the types of data and prompts being used to train models. Those things are better when they’re written and thought of by [designers] who understand where the user is coming from. [Design teams working with data teams] are getting much better results than the [teams] that are working in a vacuum.” - Paz Perez (35:19)
    •  

      Links
      • Milly Barker’s LinkedIn post
      • Greg Nudelman’s Value Matrix Article
      • Greg Nudelman website 
      • Paz Perez on Medium
      • Paz Perez on LinkedIn
      • Simon Landry LinkedIn
      • ...more
        View all episodesView all episodes
        Download on the App Store

        Experiencing Data w/ Brian T. O’Neill  (UX for AI Data Products, SAAS Analytics, Data Product Management)By Brian T. O’Neill from Designing for Analytics

        • 5
        • 5
        • 5
        • 5
        • 5

        5

        39 ratings


        More shows like Experiencing Data w/ Brian T. O’Neill (UX for AI Data Products, SAAS Analytics, Data Product Management)

        View all
        Software Engineering Radio - the podcast for professional software developers by se-radio@computer.org

        Software Engineering Radio - the podcast for professional software developers

        265 Listeners

        HBR IdeaCast by Harvard Business Review

        HBR IdeaCast

        237 Listeners

        a16z Podcast by Andreessen Horowitz

        a16z Podcast

        1,008 Listeners

        Data Skeptic by Kyle Polich

        Data Skeptic

        475 Listeners

        UI Breakfast: UI/UX Design and Product Strategy by Jane Portman

        UI Breakfast: UI/UX Design and Product Strategy

        134 Listeners

        Acquired by Ben Gilbert and David Rosenthal

        Acquired

        3,636 Listeners

        Odd Lots by Bloomberg

        Odd Lots

        1,683 Listeners

        The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) by Sam Charrington

        The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

        439 Listeners

        Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

        Super Data Science: ML & AI Podcast with Jon Krohn

        295 Listeners

        Data Engineering Podcast by Tobias Macey

        Data Engineering Podcast

        139 Listeners

        Masters of Scale by WaitWhat

        Masters of Scale

        3,986 Listeners

        DataFramed by DataCamp

        DataFramed

        266 Listeners

        Practical AI by Practical AI LLC

        Practical AI

        196 Listeners

        Machine Learning Street Talk (MLST) by Machine Learning Street Talk (MLST)

        Machine Learning Street Talk (MLST)

        92 Listeners

        Product Thinking by Melissa Perri

        Product Thinking

        142 Listeners