Just Now Possible

Turning Disruption into Opportunity: The Stack Overflow AI Story with Ellen Brandenberger

Listen Later
Guest:

Ellen Brandenburger – Product leader and coach; former head of product at Chegg Skills and Stack Overflow’s data licensing team.

What we cover in this episode:
  • How Ellen joined Stack Overflow just two weeks before ChatGPT launched, reshaping the company’s future overnight
  • The creation of Overflow AI: a team tasked with exploring “what’s just now possible” for developers
  • Four iterations of conversational search:
  • V1: a chat UI on top of keyword search
  • V2: semantic search to handle natural questions
  • V3: fallback to GPT-4 for gaps in Stack Overflow’s corpus
  • V4: adding RAG for attribution and transparency
  • Why attribution and transparency were critical for developer trust
  • How the team used simple spreadsheets and subject-matter experts to evaluate answer accuracy, relevance, and completeness
  • Why Stack decided to sunset conversational search despite heavy investment—what they learned and why it wasn’t wasted
  • The pivot to data licensing: how Stack Overflow leveraged its 14M+ Q&A corpus to power LLM training and benchmarks
  • Building industry benchmarks with subject-matter experts to prove Stack data improved LLM accuracy and relevance
    • Key lessons:
    • Take one bite of the apple at a time—prototype, learn, iterate
    • Product in the AI era means managing probabilities, not certainties
      • Links & References:
      • Ellen Brandenburger on LinkedIn
      • The Changing State of the Internet and Related Business Models
      • ProLLM: LLM benchmarks for real-world use-cases
        • ...more
        View all episodesView all episodes
        Download on the App Store
        Just Now PossibleBy Teresa Torres