The current LLM era is the result of scaling the size of models in successive waves (and the compute to train them). It is also the result of better-than-Moore’s-Law price vs performance ratios in each new generation of Nvidia GPUs. The largest platform companies are continuing to invest in scaling as the prime driver of AI innovation.

Are they right, or will marginal returns level off soon, leaving hyperscalers with too much hardware and too few customer use cases? To find out, we talk to Microsoft CTO Kevin Scott who has led their AI strategy for the past seven years. Scott describes himself as a “short-term pessimist, long-term optimist” and he sees the scaling trend as durable for the industry and critical for the establishment of Microsoft’s AI platform.

Scott believes there will be a shift across the compute ecosystem from training to inference as the frontier models continue to improve, serving wider and more reliable use cases. He also discusses the coming business models for training data, and even what ad units might look like for autonomous agents.

Hosted by: Pat Grady and Bill Coughran, Sequoia Capital

Mentioned:

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, the 2018 Google paper that convinced Kevin that Microsoft wasn’t moving fast enough on AI. 

Dennard scaling: The scaling law that describes the proportional relationship between transistor size and power use; has not held since 2012 and is often confused with Moore’s Law.

Textbooks Are All You Need: Microsoft paper that introduces a new large language model for code, phi-1, that achieves smaller size by using higher quality “textbook” data.

GPQA and MMLU: Benchmarks for reasoning

Copilot: Microsoft product line of GPT consumer assistants from general productivity to design, vacation planning, cooking and fitness.

Devin: Autonomous AI code agent from Cognition Labs that Microsoft recently announced a partnership with.

Ray Solomonoff: Participant in the 1956 Dartmouth Summer Research Project on Artificial Intelligence that named the field; Kevin admires his prescience about the importance of probabilistic methods decades before anyone else.

00:00 - Introduction

01:20 - Kevin’s backstory

06:56 - The role of PhDs in AI engineering

09:56 - Microsoft’s AI strategy

12:40 - Highlights and lowlights

16:28 - Accelerating investments

18:38 - The OpenAI partnership

22:46 - Soon inference will dwarf training

27:56 - Will the demand/supply balance change?

30:51 - Business models for data

36:54 - The value function

39:58 - Copilots

44:47 - The 98/2 rule

49:34 - Solving zero-sum games

57:13 - Lightning round

As impressive as LLMs are, the growing consensus is that language, scale and compute won’t get us to AGI. Although many AI benchmarks have quickly achieved human-level performance, there is one eval that has barely budged since it was created in 2019.

Google researcher François Chollet wrote a paper that year defining intelligence as skill-acquisition efficiency—the ability to learn new skills as humans do, from a small number of examples. To make it testable he proposed a new benchmark, the Abstraction and Reasoning Corpus (ARC), designed to be easy for humans, but hard for AI. Notably, it doesn’t rely on language.

Zapier co-founder Mike Knoop read Chollet’s paper as the LLM wave was rising. He worked quickly to integrate generative AI into Zapier’s product, but kept coming back to the lack of progress on the ARC benchmark. In June, Knoop and Chollet launched the ARC Prize, a public competition offering more than $1M to beat and open-source a solution to the ARC-AGI eval.

In this episode Mike talks about the new ideas required to solve ARC, shares updates from the first two weeks of the competition, and shares why he’s excited for AGI systems that can innovate alongside humans.

Hosted by: Sonya Huang and Pat Grady, Sequoia Capital 

Mentioned:

(00:00) Introduction

(01:51) AI at Zapier

(08:31) What is ARC AGI?

(13:25) What does it mean to efficiently acquire a new skill?

(19:03) What approaches will succeed?

(21:11) A little bit of a different shape

(25:59) The role of code generation and program synthesis

(29:11) What types of people are working on this?

(31:45) Trying to prove you wrong

(34:50) Where are the big labs?

(38:21) The world post-AGI

(42:51) When will we cross 85% on ARC AGI?

(46:12) Will LLMs be part of the solution?

(50:13) Lightning round

Archimedes said that with a large enough lever, you can move the world. For decades, software engineering has been that lever. And now, AI is compounding that lever. How will we use AI to apply 100 or 1000x leverage to the greatest lever to move the world?

Matan Grinberg and Eno Reyes, co-founders of Factory, have chosen to do things differently than many of their peers in this white-hot space. They sell a fleet of “Droids,” purpose-built dev agents which accomplish different tasks in the software development lifecycle (like code review, testing, pull requests or writing code). Rather than training their own foundation model, their approach is to build something useful for engineering orgs today on top of the rapidly improving models, aligning with the developer and evolving with them. 

Matan and Eno are optimistic about the effects of autonomy in software development and on building a company in the application layer. Their advice to founders, “The only way you can win is by executing faster and being more obsessed.”

Hosted by: Sonya Huang and Pat Grady, Sequoia Capital 

Mentioned: 

Transcript: https://www.sequoiacap.com/podcast/training-data-factory/

00:00 Introduction

01:36 Personal backgrounds

10:54 The compound lever

12:41 What is Factory? 

16:29 Cognitive architectures 

21:13 800 engineers at OpenAI are working on my margins 

24:00 Jeff Dean doesn't understand your code base

25:40 Individual dev productivity vs system-wide optimization 

30:04 Results: Factory in action 

32:54 Learnings along the way 

35:36 Fully autonomous Jeff Deans

37:56 Beacons of the upcoming age

40:04 How far are we? 

43:02 Competition 

45:32 Lightning round

49:34 Bonus round: Factory's SWE-bench results

Last year, AutoGPT and Baby AGI captured our imaginations—agents quickly became the buzzword of the day…and then things went quiet. AutoGPT and Baby AGI may have marked a peak in the hype cycle, but this year has seen a wave of agentic breakouts on the product side, from Klarna’s customer support AI to Cognition’s Devin, etc.

Harrison Chase of LangChain is focused on enabling the orchestration layer for agents. In this conversation, he explains what’s changed that’s allowing agents to improve performance and find traction. 

Harrison shares what he’s optimistic about, where he sees promise for agents vs. what he thinks will be trained into models themselves, and discusses novel kinds of UX that he imagines might transform how we experience agents in the future.     

Hosted by: Sonya Huang and Pat Grady, Sequoia Capital

Mentioned: 

Transcript: https://www.sequoiacap.com/podcast/training-data-harrison-chase/

00:00 Introduction

01:21 What are agents? 

05:00 What is LangChain’s role in the agent ecosystem?

11:13 What is a cognitive architecture? 

13:20 Is bespoke and hard coded the way the world is going, or a stop gap?

18:48 Focus on what makes your beer taste better

20:37 So what? 

22:20 Where are agents getting traction?

25:35 Reflection, chain of thought, other techniques?

30:42 UX can influence the effectiveness of the architecture

35:30 What’s out of scope?

38:04 Fine tuning vs prompting?

42:17 Existing observability tools for LLMs vs needing a new architecture/approach

45:38 Lightning round

Join us as we train our neural nets on the theme of the century: AI. Sequoia Capital partners Sonya Huang and Pat Grady host conversations with leading AI builders and researchers to ask critical questions and develop a deeper understanding of the evolving technologies and their implications for technology, business and society.

The content of this podcast does not constitute investment advice, an offer to provide investment advisory services, or an offer to sell or solicitation of an offer to buy an interest in any investment fund.