Inference Time Tactics

Benchmarking Generalization: How AI Learns Beyond Training Data

Listen Later

In this episode of Inference Time Tactics, Rob and Cooper from Neurometric sit down with Yash Sharma, an AI researcher whose work is reshaping how we understand model generalization. Yash recently completed his PhD at the Max Planck Institute for Intelligent Systems and has held research roles at Google Brain, Meta AI, Amazon, Borealis AI, and IBM Research. His studies on compositional generalization, adversarial robustness, and long-tail benchmarks reveal when and why models succeed—or fail—at reasoning beyond their training data.

If you’re designing inference-time systems, building agents that need reliability, or just want to understand what “generalization” actually means in practice, this conversation bridges deep theory with actionable insight—clear, technical, and strategically grounded.

Key Topics
  • What it really means for AI systems to generalize beyond their training data
  • Why large language models still fail in novel or unpredictable scenarios
  • How inference-time compute can both amplify and reveal generalization limits
  • What these limits mean for building reliable, agentic AI systems
  • How to benchmark generalization in real-world settings
  • Yash’s “Let It Wag!” benchmark for testing long-tail and under-represented concepts
  • Why genuine scientific breakthroughs (like curing cancer) require more than scaling test-time compute

    • Connect with Yash Sharma:

    • Yash Sharma
    • Let It Wag! Benchmark
    • Paper: Pretraining Frequency Predicts Compositional Generalization of CLIP (NeurIPS 2024 Workshop)

      • Connect with Neurometric:

      Website: https://www.neurometric.ai/ 

      Substack: https://neurometric.substack.com/ 

      X: https://x.com/neurometric/ 

      Bluesky: https://bsky.app/profile/neurometric.bsky.social

       

      Rob May

      https://x.com/robmay 

      https://www.linkedin.com/in/robmay

       

      Calvin Cooper

      https://x.com/cooper_nyc_ 

      https://www.linkedin.com/in/coopernyc

      ...more
      View all episodesView all episodes
      Download on the App Store
      Inference Time TacticsBy NeuroMetric AI