Training Data

AI researcher Jim Fan has had a charmed career. He was OpenAI’s first intern before he did his PhD at Stanford with “godmother of AI,” Fei-Fei Li. He graduated into a research scientist position at Nvidia and now leads its Embodied AI “GEAR” group. The lab’s current work spans foundation models for humanoid robots to agents for virtual worlds.

Jim describes a three-pronged data strategy for robotics, combining internet-scale data, simulation data and real world robot data. He believes that in the next few years it will be possible to create a “foundation agent” that can generalize across skills, embodiments and realities—both physical and virtual. He also supports Jensen Huang’s idea that “Everything that moves will eventually be autonomous.”

Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital

Mentioned in this episode:

00:00 Introduction

01:35 Jim’s journey to embodied intelligence

04:53 The GEAR Group

07:32 Three kinds of data for robotics

10:32 A GPT-3 moment for robotics

16:05 Choosing the humanoid robot form factor

19:37 Specialized generalists

21:59 GR00T gets its own chip

23:35 Eureka and Issac Sim

25:23 Why now for robotics?

28:53 Exploring virtual worlds

36:28 Implications for games

39:13 Is the virtual world in service of the physical world?

42:10 Alternative architectures to Transformers

44:15 Lightning round

There’s a new archetype in Silicon Valley, the AI researcher turned founder. Instead of tinkering in a garage they write papers that earn them the right to collaborate with cutting-edge labs until they break out and start their own.

This is the story of wunderkind Eric Steinberger, the founder and CEO of Magic.dev. Eric came to programming through his obsession with AI and caught the attention of DeepMind researchers as a high school student. In 2022 he realized that AGI was closer than he had previously thought and started Magic to automate the software engineering necessary to get there. Among his counterintuitive ideas are the need to train proprietary large models, that value will not accrue in the application layer and that the best agents will manage themselves. Eric also talks about Magic’s recent 100M token context window model and the HashHop eval they’re open sourcing.

Hosted by: Sonya Huang, Sequoia Capital

Mentioned in this episode:

00:00 - Introduction

01:39 - Vienna-born wunderkind

04:56 - Working with Noam Brown

8:00 - “I can do two things. I cannot do three.”

10:37 - AGI to-do list

13:27 - Advice for young researchers

20:35 - Reading every paper voraciously

23:06 - The army of Noams

26:46 - The leaps still needed in research

29:59 - What is Magic?

36:12 - Competing against the 800-pound gorillas

38:21 - Ideal team size for researchers

40:10 - AI that feels like a colleague

44:30 - Lightning round

47:50 - Bonus round: 200M token context announcement

On Training Data, we learn from innovators pushing forward the frontier of AI’s capabilities. Today we’re bringing you something different. It’s the story of a company currently implementing AI at scale in the enterprise, and how it was built from a bootstrapped idea in the pre-AI era to a 150 billion dollar market cap giant. 

It’s the Season 2 premiere of Sequoia’s other podcast, Crucible Moments, where we hear from the founders and leaders of some legendary companies about the crossroads and inflection points that shaped their journeys. In this episode, you’ll hear from Fred Luddy and Frank Slootman about building and scaling ServiceNow. Listen to Crucible Moments wherever you get your podcasts or go to:

Spotify: https://open.spotify.com/show/40bWCUSan0boCn0GZJNpPn

Apple: https://podcasts.apple.com/us/podcast/crucible-moments/id1705282398

Hosted by: Roelof Botha, Sequoia Capital

Transcript: https://www.sequoiacap.com/podcast/crucible-moments-servicenow/

Customer service is hands down the first killer app of generative AI for businesses. The reasons are simple: the costs of existing solutions are so high, the satisfaction so low and the margin for ROI so wide. But trusting your interactions with customers to hallucination-prone LLMs can be daunting.

Enter Sierra. Co-founder Clay Bavor walks us through the sophisticated engineering challenges his team solved along the way to delivering AI agents for all aspects of the customer experience that are delightful, safe and reliable—and being deployed widely by Sierra’s customers. The Company’s AgentOS enables businesses to create branded AI agents to interact with customers, follow nuanced policies and even handle customer retention and upsell. Clay describes how companies can capture their brand voice, values and internal processes to create AI agents that truly represent the business.

Hosted by: Ravi Gupta and Pat Grady, Sequoia Capital

Mentioned in this episode:

00:00:00 Introduction

00:01:21 Clay’s background

00:03:20 Google before the ChatGPT moment

00:07:31 What is Sierra?

00:12:03 What’s possible now that wasn’t possible 18 months ago?

00:17:11 AgentOS

00:23:45 The solution to many problems with AI is more AI

00:28:37 𝛕-bench

00:33:19 Engineering task vs research task

00:37:27 What tasks can you trust an agent with now?

00:43:21 What metrics will move?

00:46:22 The reality of deploying AI to customers today

00:53:33 The experience manager

01:03:54 Outcome-based pricing

01:05:55 Lightning Round

After AlphaGo beat Lee Sedol, a young mechanical engineer at Google thought of another game reinforcement learning could win: energy optimization at data centers. Jim Gao convinced his bosses at the Google data center team to let him work with the DeepMind team to try. The initial pilot resulted in a 40% energy savings and led he and his co-founders to start Phaidra to turn this technology into a product.

Jim discusses the challenges of AI readiness in industrial settings and how we have to build on top of the control systems of the 70s and 80s to achieve the promise of the Fourth Industrial Revolution. He believes this new world of self-learning systems and self-improving infrastructure is a key factor in addressing global climate change.

Hosted by: Sonya Huang and Pat Grady, Sequoia Capital 

Mentioned in this episode:

In the first wave of the generative AI revolution, startups and enterprises built on top of the best closed-source models available, mostly from OpenAI. The AI customer journey moves from training to inference, and as these first products find PMF, many are hitting a wall on latency and cost.

Fireworks Founder and CEO Lin Qiao led the PyTorch team at Meta that rebuilt the whole stack to meet the complex needs of the world’s largest B2C company. Meta moved PyTorch to its own non-profit foundation in 2022 and Lin started Fireworks with the mission to compress the timeframe of training and inference and democratize access to GenAI beyond the hyperscalers to let a diversity of AI applications thrive.

Lin predicts when open and closed source models will converge and reveals her goal to build simple API access to the totality of knowledge.

Hosted by: Sonya Huang and Pat Grady, Sequoia Capital 

Mentioned in this episode:

 00:00 - Introduction

02:01 - What is Fireworks?

02:48 - Leading Pytorch

05:01 - What do researchers like about PyTorch?

07:50 - How Fireworks compares to open source

10:38 - Simplicity scales

12:51 - From training to inference

17:46 - Will open and closed source converge?

22:18 - Can you match OpenAI on the Fireworks stack?

26:53 - What is your vision for the Fireworks platform?

31:17 - Competition for Nvidia?

32:47 - Are returns to scale starting to slow down?

34:28 - Competition

36:32 - Lightning round

GithHub invented collaborative coding and in the process changed how open source projects, startups and eventually enterprises write code. GitHub Copilot is the first blockbuster product built on top of OpenAI’s GPT models. It now accounts for more than 40 percent of GitHub revenue growth for an annual revenue run rate of $2 billion. Copilot itself is already a larger business than all of GitHub was when Microsoft acquired it in 2018.

We talk to CEO Thomas Dohmke about how a small team at GitHub built on top of GPT-3 and quickly created a product that developers love—and can’t live without. Thomas describes how the product has grown from simple autocomplete to a fully featured workspace for enterprise teams. He also believes that tools like Copilot will bring the power of coding to a billion developers by 2030.

Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital 

Mentioned in this episode:

00:00:00 - Introduction

00:01:18 - Getting started with code

00:03:43 - Microsoft’s acquisition of GitHub

00:11:40 - Evolving Copilot beyond autocomplete

00:14:18 - In hindsight, you can always move faster

00:15:56 - Building on top of OpenAI

00:20:21 - The latest metrics

00:22:11 - The surprise of Copilot’s impact

00:25:11 - Teaching kids to code in the age of Copilot

00:26:38 - The momentum mindset

00:29:46 - Agents vs Copilots

00:32:06 - The Roadmap

00:37:31 - Making maintaining software easier

00:38:48 - The creative new world

00:42:38 - The AI 10x software engineer

00:45:12 - Creativity and systems engineering in AI

00:48:55 - What about COBOL?

00:50:23 - Will GitHub build its own models?

00:57:19 - Rapid incubation at GitHub Next

00:59:21 - The future of AI?

01:03:18 - Advice for founders

01:05:08 - Lightning round

As head of Product Management for Generative AI at Meta, Joe Spisak leads the team behind Llama, which just released the new 3.1 405B model. We spoke with Joe just two days after the model’s release to ask what’s new, what it enables, and how Meta sees the role of open source in the AI ecosystem.

Joe shares that where Llama 3.1 405B really focused is on pushing scale (it was trained on 15 trillion tokens using 16,000 GPUs) and he’s excited about the zero-shot tool use it will enable, as well as its role in distillation and generating synthetic data to teach smaller models. He tells us why he thinks even frontier models will ultimately commoditize—and why that’s a good thing for the startup ecosystem.

Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital 

Mentioned in this episode: 

Llama 3.1 405B paper

Open Source AI Is the Way Forward: Mark Zuckerberg essay released with Llama 3.1.

Mistral Large 2

The Bitter Lesson by Rich Sutton

00:00 Introduction

01:28 The Llama 3.1 405B launch

05:02 The open source license

07:01 What's in it for Meta?

10:19 Why not open source?

11:16 Will frontier models commoditize?

12:41 What about startups?

16:29 The Mistral team

19:36 Are all frontier strategies comparable?

22:38 Is model development becoming more like software development?

26:34 Agentic reasoning

29:09 What future levers will unlock reasoning?

31:20 Will coding and math lead to unlocks?

33:09 Small models

34:08 7X more data

37:36 Are we going to hit a wall?

39:49 Lightning round

In February, Sebastian Siemiatkowski boldly announced that Klarna’s new OpenAI-powered assistant handled two thirds of the Swedish fintech’s customer service chats in its first month. Not only were customer satisfaction metrics better, but by replacing 700 full-time contractors the bottom line impact is projected to be $40M. Since then, every company we talk to wants to know, “How do we get the Klarna customer support thing?”

Co-founder and CEO Sebastian Siemiatkowski tells us how the Klarna team shipped this new product in record time—and how embracing AI internally with an experimental mindset is transforming the company. He discusses how AI development is proliferating inside the company, from customer support to marketing to internal knowledge to customer-facing experiences. 

Sebastian also reflects on the impacts of AI on employment, society, and the arts while encouraging lawmakers to be open minded about the benefits.

Hosted by: Sonya Huang and Pat Grady, Sequoia Capital 

Mentioned in this episode:

DeepL: Language translation app that Sebastian says makes 10,000 translators in Brussels redundant

The Klarna brand: The offbeat optimism that the company is now augmenting with AI

Neo4j: The graph database management system that Klarna is using to build Kiki, their internal knowledge base

00:00 Introduction

01:57 Klarna’s business

03:00 Pitching OpenAI

08:51 How we built this

10:46 Will Klara ever completely replace its CS team with AI?

14:22 The benefits

17:25 If you had a policy magic wand…

21:12 What jobs will be most affected by AI?

23:58 How about marketing?

27:55 How creative are LLMs?

30:11 Klarna’s knowledge graph, Kiki

33:10 Reducing the number of enterprise systems

35:24 Build vs buy?

39:59 What’s next for Klarna with AI?

48:48 Lightning round

LLMs are democratizing digital intelligence, but we’re all waiting for AI agents to take this to the next level by planning tasks and executing actions to actually transform the way we work and live our lives. 

Yet despite incredible hype around AI agents, we’re still far from that “tipping point” with best in class models today. As one measure: coding agents are now scoring in the high-teens % on the SWE-bench benchmark for resolving GitHub issues, which far exceeds the previous unassisted baseline of 2% and the assisted baseline of 5%, but we’ve still got a long way to go.

Why is that? What do we need to truly unlock agentic capability for LLMs? What can we learn from researchers who have built both the most powerful agents in the world, like AlphaGo, and the most powerful LLMs in the world? 

To find out, we’re talking to Misha Laskin, former research scientist at DeepMind. Misha is embarking on his vision to build the best agent models by bringing the search capabilities of RL together with LLMs at his new company, Reflection AI. He and his cofounder Ioannis Antonoglou, co-creator of AlphaGo and AlphaZero and RLHF lead for Gemini, are leveraging their unique insights to train the most reliable models for developers building agentic workflows.

Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital 

00:00 Introduction

01:11 Leaving Russia, discovering science

10:01 Getting into AI with Ioannis Antonoglou

15:54 Reflection AI and agents

25:41 The current state of Ai agents

29:17 AlphaGo, AlphaZero and Gemini

32:58 LLMs don’t have a ground truth reward

37:53 The importance of post-training

44:12 Task categories for agents

45:54 Attracting talent

50:52 How far away are capable agents?

56:01 Lightning round

Mentioned: