Latent Space

From Berkeley robotics and OpenAI’s 2017 Dota-era internship to shipping RL breakthroughs on GPT-4o, o1, and o3, and now leading model development at Cursor, Ashvin Nair has done it all. We caught up with Ashvin at NeurIPS 2025 to dig into the inside story of OpenAI’s reasoning team (spoiler: it went from a dozen people to 300+), why IOI Gold felt reachable in 2022 but somehow didn’t change the world when o1 actually achieved it, how RL doesn’t generalize beyond the training distribution (and why that means you need to bring economically useful tasks into distribution by co-designing products and models), the deeper lessons from the RL research era (2017–2022) and why most of it didn’t pan out because the community overfitted to benchmarks, how Cursor is uniquely positioned to do continual learning at scale with policy updates every two hours and product-model co-design that keeps engineers in the loop instead of context-switching into ADHD hell, and his bet that the next paradigm shift is continual learning with infinite memory—where models experience something once (a bug, a mistake, a user pattern) and never forget it, storing millions of deployment tokens in weights without overloading capacity.

We discuss:

• Ashvin’s path: Berkeley robotics PhD → OpenAI 2017 intern (Dota era) → o1/o3 reasoning team → Cursor ML lead in three months

• Why robotics people are the most grounded at NeurIPS (they work with the real world) and simulation people are the most unhinged (Lex Fridman’s take)

• The IOI Gold paradox: “If you told me we’d achieve IOI Gold in 2022, I’d assume we could all go on vacation—AI solved, no point working anymore. But life is still the same.”

• The RL research era (2017–2022) and why most of it didn’t pan out: overfitting to benchmarks, too many implicit knobs to tune, and the community rewarding complex ideas over simple ones that generalize

• Inside the o1 origin story: a dozen people, conviction from Ilya and Jakob Pachocki that RL would work, small-scale prototypes producing “surprisingly accurate reasoning traces” on math, and first-principles belief that scaled

• The reasoning team grew from ~12 to 300+ people as o1 became a product and safety, tooling, and deployment scaled up

• Why Cursor is uniquely positioned for continual learning: policy updates every two hours (online RL on tab), product and ML sitting next to each other, and the entire software engineering workflow (code, logs, debugging, DataDog) living in the product

• Composer as the start of product-model co-design: smart enough to use, fast enough to stay in the loop, and built by a 20–25 person ML team with high-taste co-founders who code daily

• The next paradigm shift: continual learning with infinite memory—models that experience something once (a bug, a user mistake) and store it in weights forever, learning from millions of deployment tokens without overloading capacity (trillions of pretraining tokens = plenty of room)

• Why off-policy RL is unstable (Ashvin’s favorite interview question) and why Cursor does two-day work trials instead of whiteboard interviews

• The vision: automate software engineering as a process (not just answering prompts), co-design products so the entire workflow (write code, check logs, debug, iterate) is in-distribution for RL, and make models that never make the same mistake twice

—

Ashvin Nair

• Cursor: https://cursor.com

• X: https://x.com/ashvinnair_

00:00:00 Introduction: From Robotics to Cursor via OpenAI
00:01:58 The Robotics to LLM Agent Transition: Why Code Won
00:09:11 RL Research Winter and Academic Overfitting
00:11:45 The Scaling Era and Moving Goalposts: IOI Gold Doesn't Mean AGI
00:21:30 OpenAI's Reasoning Journey: From Codex to O1
00:20:03 The Blip: Thanksgiving 2023 and OpenAI Governance
00:22:39 RL for Reasoning: The O-Series Conviction and Scaling
00:25:47 O1 to O3: Smooth Internal Progress vs External Hype Cycles
00:33:07 Why Cursor: Co-Designing Products and Models for Real Work
00:34:14 Composer and the Future: Online Learning Every Two Hours
00:35:15 Continual Learning: The Missing Paradigm Shift
00:44:00 Hiring at Cursor and Why Off-Policy RL is Unstable

From investing through the modern data stack era (DBT, Fivetran, and the analytics explosion) to now investing at the frontier of AI infrastructure and applications at Amplify Partners, Sarah Catanzaro has spent years at the intersection of data, compute, and intelligence—watching categories emerge, merge, and occasionally disappoint. We caught up with Sarah live at NeurIPS 2025 to dig into the state of AI startups heading into 2026: why $100M+ seed rounds with no near-term roadmap are now the norm (and why that terrifies her), what the DBT-Fivetran merger really signals about the modern data stack (spoiler: it’s not dead, just ready for IPO), how frontier labs are using DBT and Fivetran to manage training data and agent analytics at scale, why data catalogs failed as standalone products but might succeed as metadata services for agents, the consumerization of AI and why personalization (memory, continual learning, K-factor) is the 2026 unlock for retention and growth, why she thinks RL environments are a fad and real-world logs beat synthetic clones every time, and her thesis for the most exciting AI startups: companies that marry hard research problems (RAG, rule-following, continual learning) with killer applications that were simply impossible before.

We discuss:

• The DBT-Fivetran merger: not the death of the modern data stack, but a path to IPO scale (targeting $600M+ combined revenue) and a signal that both companies were already winning their categories

• How frontier labs use data infrastructure: DBT and Fivetran for training data curation, agent analytics, and managing increasingly complex interactions—plus the rise of transactional databases (RocksDB) and efficient data loading (Vortex) for GPU-bound workloads

• Why data catalogs failed: built for humans when they should have been built for machines, focused on discoverability when the real opportunity was governance, and ultimately subsumed as features inside Snowflake, DBT, and Fivetran

• The $100M+ seed phenomenon: raising massive rounds at billion-dollar valuations with no 6-month roadmap, seven-day decision windows, and founders optimizing for signal (”we’re a unicorn”) over partnership or dilution discipline

• Why world models are overhyped but underspecified: three competing definitions, unclear generalization across use cases (video games ≠ robotics ≠ autonomous driving), and a research problem masquerading as a product category

• The 2026 theme: consumerization of AI via personalization—memory management, continual learning, and solving retention/churn by making products learn skills, preferences, and adapt as the world changes (not just storing facts in cursor rules)

• Why RL environments are a fad: labs are paying 7–8 figures for synthetic clones when real-world logs, traces, and user activity (à la Cursor) are richer, cheaper, and more generalizable

• Sarah’s investment thesis: research-driven applications that solve hard technical problems (RAG for Harvey, rule-following for Sierra, continual learning for the next killer app) and unlock experiences that were impossible before

• Infrastructure bets: memory, continual learning, stateful inference, and the systems challenges of loading/unloading personalized weights at scale

• Why K-factor and growth fundamentals matter again: AI felt magical in 2023–2024, but as the magic fades, retention and virality are back—and most AI founders have never heard of K-factor

—

Sarah Catanzaro

• X: https://x.com/sarahcat21

• Amplify Partners: https://amplifypartners.com/

Where to find Latent Space

• X: https://x.com/latentspacepod

00:00:00 Introduction: Sarah Catanzaro's Journey from Data to AI
00:01:02 The DBT-Fivetran Merger: Not the End of the Modern Data Stack
00:05:26 Data Catalogs and What Went Wrong
00:08:16 Data Infrastructure at AI Labs: Surprising Insights
00:10:13 The Crazy Funding Environment of 2024-2025
00:17:18 World Models: Hype, Confusion, and Market Potential
00:18:59 Memory Management and Continual Learning: The Next Frontier
00:23:27 Agent Environments: Just a Fad?
00:25:48 The Perfect AI Startup: Research Meets Application
00:28:02 Closing Thoughts and Where to Find Sarah

One year ago, Anthropic launched the Model Context Protocol (MCP)—a simple, open standard to connect AI applications to the data and tools they need. Today, MCP has exploded from a local-only experiment into the de facto protocol for agentic systems, adopted by OpenAI, Microsoft, Google, Block, and hundreds of enterprises building internal agents at scale. And now, MCP is joining the newly formed Agentic AI Foundation (AAIF) under the Linux Foundation, alongside Block’s Goose coding agent, with founding members spanning the biggest names in AI and cloud infrastructure.

We sat down with David Soria Parra (MCP lead, Anthropic), Nick Cooper (OpenAI), Brad Howes (Block / Goose), and Jim Zemlin (Linux Foundation CEO) to dig into the one-year journey of MCP—from Thanksgiving hacking sessions and the first remote authentication spec to long-running tasks, MCP Apps, and the rise of agent-to-agent communication—and the behind-the-scenes story of how three competitive AI labs came together to donate their protocols and agents to a neutral foundation, why enterprises are deploying MCP servers faster than anyone expected (most of it invisible, internal, and at massive scale), what it takes to design a protocol that works for both simple tool calls and complex multi-agent orchestration, how the foundation will balance taste-making (curating meaningful projects) with openness (avoiding vendor lock-in), and the 2025 vision: MCP as the communication layer for asynchronous, long-running agents that work while you sleep, discover and install their own tools, and unlock the next order of magnitude in AI productivity.

We discuss:

• The one-year MCP journey: from local stdio servers to remote HTTP streaming, OAuth 2.1 authentication (and the enterprise lessons learned), long-running tasks, and MCP Apps (iframes for richer UI)

• Why MCP adoption is exploding internally at enterprises: invisible, internal servers connecting agents to Slack, Linear, proprietary data, and compliance-heavy workflows (financial services, healthcare)

• The authentication evolution: separating resource servers from identity providers, dynamic client registration, and why the March spec wasn’t enterprise-ready (and how June fixed it)

• How Anthropic dogfoods MCP: internal gateway, custom servers for Slack summaries and employee surveys, and why MCP was born from “how do I scale dev tooling faster than the company grows?”

• Tasks: the new primitive for long-running, asynchronous agent operations—why tools aren’t enough, how tasks enable deep research and agent-to-agent handoffs, and the design choice to make tasks a “container” (not just async tools)

• MCP Apps: why iframes, how to handle styles and branding, seat selection and shopping UIs as the killer use case, and the collaboration with OpenAI to build a common standard

• The registry problem: official registry vs. curated sub-registries (Smithery, GitHub), trust levels, model-driven discovery, and why MCP needs “npm for agents” (but with signatures and HIPAA/financial compliance)

• The founding story of AAIF: how Anthropic, OpenAI, and Block came together (spoiler: they didn’t know each other were talking to Linux Foundation), why neutrality matters, and how Jim Zemlin has never seen this much day-one inbound interest in 22 years

—

David Soria Parra (Anthropic / MCP)

• MCP: https://modelcontextprotocol.io

• https://uk.linkedin.com/in/david-soria-parra-4a78b3a

• https://x.com/dsp_

Nick Cooper (OpenAI)

• X: https://x.com/nicoaicopr

Brad Howes (Block / Goose)

• Goose: https://github.com/block/goose

Jim Zemlin (Linux Foundation)

• LinkedIn: https://www.linkedin.com/in/zemlin/

Agentic AI Foundation

• https://agenticai.foundation

00:00:00 Introduction: MCP's First Year and Foundation Launch
00:01:17 MCP's Journey: From Launch to Industry Standard
00:02:06 Protocol Evolution: Remote Servers and Authentication
00:08:52 Enterprise Authentication and Financial Services
00:11:42 Transport Layer Challenges: HTTP Streaming and Scalability
00:15:37 Standards Development: Collaboration with Tech Giants
00:34:27 Long-Running Tasks: The Future of Async Agents
00:30:41 Discovery and Registries: Building the MCP Ecosystem
00:30:54 MCP Apps and UI: Beyond Text Interfaces
00:26:55 Internal Adoption: How Anthropic Uses MCP
00:23:15 Skills vs MCP: Complementary Not Competing
00:36:16 Community Events and Enterprise Learnings
01:03:31 Foundation Formation: Why Now and Why Together
01:07:38 Linux Foundation Partnership: Structure and Governance
01:11:13 Goose as Reference Implementation
01:17:28 Principles Over Roadmaps: Composability and Quality
01:21:02 Foundation Value Proposition: Why Contribute
01:27:49 Practical Investments: Events, Tools, and Community
01:34:58 Looking Ahead: Async Agents and Real Impact

Note: Steve and Gene’s talk on Vibe Coding and the post IDE world was one of the top talks of AIE CODE:

From building legendary platforms at Google and Amazon to authoring one of the most influential essays on AI-powered development (Revenge of the Junior Developer, quoted by Dario Amodei himself), Steve Yegge has spent decades at the frontier of software engineering—and now he’s leading the charge into what he calls the “factory farming” era of code. After stints at SourceGraph and building Beads (a purely vibe-coded issue tracker with tens of thousands of users), Steve co-authored The Vibe Coding Book and is now building VC (VibeCoder), an agent orchestration dashboard designed to move developers from writing code to managing fleets of AI agents that coordinate, parallelize, and ship features while you sleep.

We sat down with Steve at AI Engineer Summit to dig into why Claude Code, Cursor, and the entire 2024 stack are already obsolete, what it actually takes to trust an agent after 2,000 hours of practice (hint: they will delete your production database if you anthropomorphize them), why the real skill is no longer writing code but orchestrating agents like a NASCAR pit crew, how merging has become the new wall that every 10x-productive team is hitting (and why one company’s solution is literally “one engineer per repo”), the rise of multi-agent workflows where agents reserve files, message each other via MCP, and coordinate like a little village, why Steve believes if you’re still using an IDE to write code by January 1st, you’re a bad engineer, how the 12–15 year experience bracket is the most resistant demographic (and why their identity is tied to obsolete workflows), the hidden chaos inside OpenAI, Anthropic, and Google as they scale at breakneck speed, why rewriting from scratch is now faster than refactoring for a growing class of codebases, and his 2025 prediction: we’re moving from subsistence agriculture to John Deere-scale factory farming of code, and the Luddite backlash is only just beginning.

We discuss:

• Why Claude Code, Cursor, and agentic coding tools are already last year’s tech—and what comes next: agent orchestration dashboards where you manage fleets, not write lines

• The 2,000-hour rule: why it takes a full year of daily use before you can predict what an LLM will do, and why trust = predictability, not capability

• Steve’s hot take: if you’re still using an IDE to develop code by January 1st, 2025, you’re a bad engineer—because the abstraction layer has moved from models to full-stack agents

• The demographic most resistant to vibe coding: 12–15 years of experience, senior engineers whose identity is tied to the way they work today, and why they’re about to become the interns

• Why anthropomorphizing LLMs is the biggest mistake: the “hot hand” fallacy, agent amnesia, and how Steve’s agent once locked him out of prod by changing his password to “fix” a problem

• Should kids learn to code? Steve’s take: learn to vibe code—understand functions, classes, architecture, and capabilities in a language-neutral way, but skip the syntax

• The 2025 vision: “factory farming of code” where orchestrators run Cloud Code, scrub output, plan-implement-review-test in loops, and unlock programming for non-programmers at scale

—

Steve Yegge

• X: https://x.com/steve_yegge

• Substack (Stevie’s Tech Talks): https://steve-yegge.medium.com/

• GitHub (VC / VibeCoder): https://github.com/yegge-labs

Where to find Latent Space

• X: https://x.com/latentspacepod

00:00:00 Introduction: Steve Yegge on Vibe Coding and AI Engineering
00:00:59 The Backlash: Who Resists Vibe Coding and Why
00:04:26 The 2000 Hour Rule: Building Trust with AI Coding Tools
00:03:31 The January 1st Deadline: IDEs Are Becoming Obsolete
00:02:55 10X Productivity at OpenAI: The Performance Review Problem
00:07:49 The Hot Hand Fallacy: When AI Agents Betray Your Trust
00:11:12 Claude Code Isn't It: The Need for Agent Orchestration
00:15:20 The Orchestrator Revolution: From Cloud Code to Agent Villages
00:18:46 The Merge Wall: The Biggest Unsolved Problem in AI Coding
00:26:33 Never Rewrite Your Code - Until Now: Joel Spolsky Was Wrong
00:22:43 Factory Farming Code: The John Deere Era of Software
00:29:27 Google's Gemini Turnaround and the AI Lab Chaos
00:33:20 Should Your Kids Learn to Code? The New Answer
00:34:59 Code MCP and the Gossip Rate: Latest Vibe Coding Discoveries

4 days, focused writing, feedback, peer review, framework, writing community!

I’m excited to open up 10 Dev Writers Retreat Fellowship slots to Latent Space subscribers - if you’re reading this in you…

AIE CODE is sold out, but you can watch the livestream, join AIE CODE++ in SF, or Dev Writers Retreat after NeurIPS!

We are on the record as fans of Steph Palazzolo, but we politely disagree with her coining of the term “Neolab”, adopted and boosted by returning guest Deedy Das. For AI Engineer Code Summit we used, for the first time, the term “Agent Lab”, instead.

“Agent Labs” capture an orthogonal trend to Steph’s “Neolabs that exploit new approaches to developing AI models and research that OpenAI and Anthropic have overlooked” (eg Thinking Machines, Humans&, General Intuition, SSI, Reflection and more) and instead covers Cursor ($29B), Perplexity ($20B), Cognition ($10B), Sierra ($10B), Lovable ($2B), Gamma ($2B), as well as older companies with successful AI transitions like Notion ($10B), Vercel ($9B), Glean ($7B), and Replit ($3B), as well as “Agent Labs within Model Labs” like Claude Code ($1B ARR), Codex, and Google Labs.

The problem with “NEOlab” is the same as “NEOcloud” and “MODERN Data Stack” — just being “new” isn’t a business plan or an investment thesis; everybody “Brings Your Own Context” and so agreeing there is as fruitful as building a Tower in Babel.

Agent Labs have their business plan embedded in the name: where Model Labs primarily research and sell models, Agent Labs primarily research and sell agents.

necessary disclaimer: not all agent labs will check these boxes, but I believe they will converge on these best practices over time due to PMF pull/economics.

• Product First, Model Last: The “Neolab” way was how Magic.dev raised $100m to work on their long context model. Instead, Cursor forked VSCode first and only after 2 years of understanding user needs, then worked on the model.

• Outcome-based Pricing or Marketing: When you are a Model Lab, you are on that 9-900x/year distillation grind, and with not much pricing power token-for-token with the LLM Pareto Frontier (exception for Anthropic!). People complain about and max out your $20/month a subscriptions. When you are an Agent Lab, you can charge $2000/month or per outcome, and as long as the outcomes hold true, you have much higher margins, pricing power, and even growth, because you measurably replace some subsection of human labor.

• Approach to autonomy: Model Labs typically want to take control away from the user because they prioritize hours of autonomy, which is easier to test and perhaps on the critical path for pursuing fully autonomous AGI. They emphasize lightweight harnesses (Building Effective Agents and Skills) because they work so closely with the model teams and believe that the next model upgrade might undo harness gains. Agent Labs prioritize speed, auditable/human in the loop control and multiturn interactivity, and don’t mind rewriting the harness every few months for the gains they bring from the future to today.

• Evals/metrics/prioritization: (this is a weaker point) charts for Frontier Model Lab models are often 1 dimensional because they are about pushing maximum capabilities (e.g. IMO, IOI) while ignoring cost. Agent labs focus on high volume and practical usage and typically care about the pareto frontier of intelligence/success vs cost. This is a weaker point because of course Model labs care about efficiency too, but it is typically a secondary concern.

This isn’t by any means an authoritative or even well researched diagram, but my theory is that Conway’s Law is the biggest tell — a company’s priorities are demonstrated through its resource allocation of money, people and time. While Model Labs often build Agent Labs internally, and Agent Labs do build Models, if you step back and look at the whole picture you probably get a stark difference:

Anecdotally, model labs pay “Applied AI Engineers” roughly half of what they pay "Research staff” (this ratio of course varies WIDELY given the Zuck-shaped skewness in researcher and research-engineer compensation), whereas FDEs and GTMEs can often be the most important people in an Agent Lab which basically lives to maximize learning from customers.

Another “tell” of a non-Agent lab is how willingly they “drop alpha”, aka open source agents — OpenAI recently shared their internal sales assistant, support agent, research assistant, contract data, and GTM. Vercel (which is an AI Cloud that also has an Agent Lab) also recently shared lessons from 5 of their agents, for Support to v0 to code review to leadgen to data analyst. It’s a classic tech strategy play: commoditize your complements. Model Labs will happily teach you how to build Agents for free, Agent Labs will happily abstract away the model selector in favor of task models. Model Labs don’t really sweat the B2B/enterprise needs, Agent Labs do.

A (weaker) tell about Model Labs vs Agent Labs is that acquihired founders have a high rate of leaving Model Labs (identifying recent examples is left as an exercise to the reader), whereas Agent Labs tend to boast how many former founders they have (more product oriented).

While Model (Neo)Labs like Jeff Bezos’ Project Prometheus (raising $6b) and Thinking Machines (raising another $5b!!!) are extremely capital intensive, Agent Labs have better cashflow economics, but you’ll have to wait 10-15 years to see what their exit valuations look like compared to the median Model Lab/Neolab. However we’re starting to see that Agent Labs can competitively hire vs Model Labs — recent examples including Sasha Rush and Less Wright.

Two datapoints opened my eyes to the disparity between Model Labs and Agent Labs recently. First, Epoch’s estimates that all Inference compute for OpenAI (all of ChatGPT, all of Sora, all of Codex, all of API inference, all internal usage, and so on) is only the skinny blue 28% of their compute resources:

This puts some hard numbers to the general impression that, even though everyone talks about ChatGPT and the recent product ships like Sora and Atlas, the vast majority of OpenAI’s resources are dedicated toward unpublished fundamental research, as it should be.

The second clue was the OpenAI Foundation livestream where Sam mapped out the future priorities of the company and said the words “AI Cloud”, “Third Party apps” and even quoted the Bill Gates Line for the first time in OpenAI history.

OpenAI’s destiny as an AI cloud is something we have been anticipating for the last 2 years, and have been looking for in every single DevDay from 2023 to 2024 to 2025, but Sam has never, ever, spoken so clearly about OpenAI’s intention to serve third party builders at the app layer rather than primarily compete with them by building its own Apps and having ChatGPT be a “superapp”. This makes absolute economic sense - OpenAI achieves the most scale by going DOWN the stack to building its own chips and datacenters and power sources, and is far more likely to achieve AGI that way than by going *UP* the stack.

OpenAI isn’t the only Model Lab doing this pivot to AI cloud, it is just the furthest along. We’ve recently done podcasts with both Vercel and GitHub to get familiar with their AI Cloud strategy, and former guest Rita Kozlov today just announced Cloudflare’s acquisition of Replicate (another former guest!). However, they are all likely going to be outpaced in infra spend by Anthropic, which is now closing its $350B fundraise and announced its first $50B datacenter. They have ALSO expanded and focused their Claude Developer efforts and are the most serious contender to OpenAI:

One year on from writing Why GPT Wrappers are Good, Actually, we now actually have official blessing from leading frontier labs that they are finally taking AI Engineers more seriously than they have historically been.

So, let’s recap:

• Priorities. Model Labs prioritize AGI models, Agent Labs prioritize AI agents.

  • Model Labs spend <30% of their budget on inference, applied AI engineers are paid 50-70% what research engineers get, and products get abandoned all the time: Operator, NotebookLM Audio Overviews, Scheduled Tasks, Deep Research, {other notable model branches I can’t name}. Both OpenAI and Anthropic have now signaled that they are building platforms for developers to build on rather than pursue first party apps, as they realize that being hyperscalers/”AI Clouds” and platforms is a far more scalable strategy.

    • with the notable exception of Claude Code, Codex, and Sora, and maybe Claude for Finance. Model Labs are building Agent Labs inside of them, at the same time that Agent Labs are starting to build their own models.

• Model Diversity. the number of relevant frontier Model Labs has gone UP, not down, including Chinese/US open model labs, which means 1) you can capabilitymaxx across all of them, 2) people want to pay someone to capabilitymaxx for them for the literal full time job of keeping up with the state of the art and building the best combination of model and harness to the task. Often it’s an inhouse AI czar, but as this speciality deepends, it’s Agent Labs.

  from our Deedy 2025 pod

  • Agents are Systems. The Model Selector is a Lie. Agents in the real world are bundles of (model, prompt, memories, tools, planning, orchestration, auth) so the competitive advantage of model labs in building such end to end systems declines since they are primarily differentiated in the model layer.

• Pretraining → RL. Since Alexnet (2012) and GPT-1 (2018) there is now about 7-13 years of scaling pretraining, where general knowledge and unsupervised language modeling of the average human response are mostly solved and we are close to hitting the limits of pretraining data. The RL age rewards domain focus/dedicated environments and matching/surpassing best humans.

  When Grok 4 (now known to be 3T parameters) is indicating roughly equivalent spend on posttraining as pretraining, the thesis that Jeremy Howard laid out on our pod 2 years ago is playing out; and now the Agent Labs like Cursor and Cognition can essentially start from increasingly powerful open weights models and just do continued training.

  If you pay close attention to Sasha Rush’s talk at Ray Summit, you can notice that the x axis is log scale, which, while it is not quantified, can be presumed that Cursor is now saying their post-training can largely make up the gap between the best open models and the best frontier models. How long more until they start exceeding?

This last model training piece, as well as general research on all Agent Engineering and Agent Research is why it’s ALSO important to keep the Labs half of Agent Labs - these aren’t simply “Agent Companies” or “Agent Startups” - to be feasibly frontier at what they do, they likely have to invest in more R&D and fast experimentation in the actual scientific sense, not just the tax accounting one.

The “bear case” for the distinction between Agent Labs and Model Labs being meaningful, is if the Agent Labs embedded within Model Labs grow so powerful and important that they basically create a lasting fork in the model tree.

To elaborate: until 2025, the idea of a Model Lab pursuing “AGI” meant one model that could do all things - answer MMLU, vibe code a full stack app, and also roleplay as a friend. That puts the “G” in “AGI”. Just a little over a year ago, the GPT 4o launch promised to unify audio/image/text input and audio/image/text output. The “o” of “4o” was meant to be “omnimodel”.

However, to this day, GPT5 has evaded omnimodality, and continuing issues with the GPT5 router, the persistence of gpt-5-codex vs the regular gpt-5, and Fidji Simo’s blogpost literally titled “Moving beyond one-size-fits all”, suggest that at least for now, the Model Lab vision might be shifting, at least until the next big algorithm shift.

Latent Space is growing! We are proud to announce our new AI4Science show, with new hosts Brandon and RJ, as CZI, Google, OpenAI, and others all ramp up their applied AI work on applying the coming superintelligence to solving humanity’s hardest scientific problems across biology, physics, math, and more. Please comment requests for guests and topics!

We’ll be in NYC for AIE CODE! Eng day tix sold out, grab the last few!

For San Diego NeurIPS, join us for Model Ship and the 2025 Dev Writers Retreat!

So… Priscilla and Mark dropped by the pod, their first joint appearance since Huberman 2023. We’re still pinching ourselves.

When I talk to AI Engineer friends and podcast listeners, a lot of them confess that they are intimidated by the hard sciences and as a result don’t know how to engage/fear they can’t follow along.

Don’t worry, I’ve got you. Everyone here is familiar with the “business model” of a frontier AI lab, and CZI is now all-in on Biohub as THE frontier AI Bio lab. This is not just a marketing tagline - AI is eating Biology in the same way it has conquered natural language and software.

We recently supported Lux Capital’s prestigious AI Summit in NYC and Sam Sinai noted how research that used to take 10 scientists and a year of in vivo experiments can now be done by “one scientist and a few weeks of design”. That’s a familiar story to any AI Engineer.

The Human Genome Project sequenced and open sourced 3B pairs of human DNA.

AlphaFold 3D-modeled and open sourced the structure of 200m proteins (a protein is roughly 50-1000x larger in complexity and size than a gene).

CZI’s Biohub has $1b/year to make long-horizon (10-15 year) bets on basic science:

• Image and contribute to the Human Cell Atlas: the ~billions of cells of the human body (a cell can contain billions of protein molecules, building on OpenCell) to create a unified AI-based virtual cell model to predict and understand how cells behave within the human body (“in silico” is 1000x cheaper/faster than “in vivo”)

• Invent all the necessary hardware including CryoET microscopes to do the above

• Annotate and develop models (eg FAIR’s SAM2) and pipelines to help annotate

• Developing and funding open source software incl CELLxGENE Annotate

• Researching neurodegenerative, infectious (pandemic), and rare diseases as Rare As One person with <10% treatments

• Physically colocating for interdisciplinary collaboration the AI researchers, engineers and leading cell biologists, building on the original Biohub model in Chicago, New York, and SF and Imaging Institute.

While the HGP provided the “parts list” and AlphaFold revealed the shape of those parts, CZI and Biohub will answer how all the parts work together in the complex, dynamic system of a cell — and now, 10 years after the founding of CZI, going beyond.

With the acquisition of Evoscale (the team behind ESM-3), and 10xing compute capacity to build a 10,000 GPU cluster, Biohub is now positioned to go beyond the Virtual Cell to now also study the Virtual Immune System. We REALLY, really liked how Priscilla phrased how the Immune System is the next obvious target for health:

“The clinical impact of understanding the immune system is huge. Because biology has already given us a way to keep the body healthy. And it also sometimes goes awry and causes disease with autoimmune disease, right? Yeah. It’s a very complex system that has to stay in balance. And if it goes out of balance in either direction, you get sick. It can also go into your body, and it’s a privileged system that is mobile and can go into places like your brain, your pancreas, your heart, to either do maintenance or to collect signal. That’s built in. So if we can understand this system, we can use it to keep people healthy.

We already kind of do. There’s CAR T-cells, where we reprogram T-cells to go in and fight cancer. And in our New York Biohub, we’re doing cellular engineering to say, like, hey, can you go in to this person’s heart, check if they have plaques that are causing problems, read it into your DNA, self-analyse, and then we can read out the signal as cell-free DNA and give us a binary answer, yes or no. Right? I know it sounds sci-fi. It is realistic. It is happening.

And then on the other end of understanding the balance, like, so many autoimmune diseases, MS, lupus, those are the ones, examples of ones we know. I think there are other things that are autoimmune that we don’t understand, like dementia, [where] autoimmunity can play a large role in that. And so if we can understand the fine balance that the system needs to be kept in, then we can understand the balance. Then we can actually impact a lot of the ways the human body is maintained. So I think it’s both interesting from a biology perspective and feasible to model, and probably one of the highest sort of now impact systems if we can learn how to manipulate.”

Mark: “Amazing. But it’s only one system, right?”

Priscilla: “So it’s a subset but if you’re focused on curing and preventing diseases, the immune system is a pretty important one.”

Biohub’s aim is to change the understanding of immunological science — opening the door to engineering human health, simulating immune therapies, reprogramming dysfunctional cells, and preventing diseases before they arise.

Biohub also opened up three new models on its virtual cells platform:

• VariantFormer: a model that directly translates personal genetic variations into tissue-specific gene activity patterns;

• CryoLens: an end-to-end, pre-trained, large-scale model for cryoET providing unsupervised structural similarity analysis; and

• scLDM: a new AI model that can generate realistic single-cell data in silico at unprecedented fidelity.

These complement other leading models that were recently made available, including GREmLN, a graph-aware model for understanding how gene regulatory networks govern cell behavior, and rBio, a conversational large language model designed to perform biological reasoning and make scientific knowledge more accessible, and other models, providing a comprehensive platform for AI-based research in biology (which we recently discussed in the Latent Space Paper Club!)

on YouTube and wherever the finest podcasts are sold!

• Biohub

• Alex Rives (Evolutionary Scale)

• Human Cell Atlas

• CELLxGENE

• rbio1

• TranscriptFormer

• VariantFormer

• ESM3 (Evolutionary Scale Modeling 3)

• GREmLN

• Rare As One Project

• CAR T-cell therapy

• CZI Imaging Institute

• San Francisco Biohub

• New York Biohub

• Chicago Biohub

• [00:00:00] Introduction and CZI’s 10-Year Anniversary

• [00:00:56] Learning from Bill Gates

• [00:04:05] Science vs Translation

• [00:10:45] The Power of Physical Proximity in Science

• [00:13:55] Building the Virtual Cell: From Data to Models

• [00:15:51] Microscopes, Imaging, and Converting Atoms to Bits

• [00:23:18] AI Meets Biology: The Frontier Lab Concept

• [00:27:25] How Models Can Enable More Ambitious Research

• [00:28:45] Joining Forces with Evolutionary Scale

• [00:30:15] Precision Medicine and Clinical Impact

• [00:45:17] The Virtual Immune System and Cellular Engineering

• [00:48:27] Accelerating the Timeline: What It Takes to Cure All Disease

Alessio [00:00:04]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, founder of Kernel Labs, and I’m joined by Swyx, editor of Latent Space.

swyx [00:00:10]: Hello, we’re so delighted to be in the Imaging Institute of CZI with literally C and Z. Welcome Mark and Priscilla.

Priscilla [00:00:17]: Thanks for having us. Thanks for getting nerdy.

swyx [00:00:19]: Yeah, we’re excited to do this. We so don’t often get to see this side of you, and so thank you for taking some time out to talk about this. And it’s like, you know, sort of the 10-year anniversary kind of of CZI. So I just wanted to introduce people if, you know, people have not been caught up. One of the interesting things that we found out just from talking to your teams is there is an interesting difference between how you guys started CZI and the Gates Foundation. And I heard that Bill Gates is a mentor of yours. So maybe you could tell that story of like deciding to start CZI and deciding to pursue basic science instead of translational work.

Mark [00:00:55]: Well, I mean, I think one of the core things for us with CZI was just getting... We got some advice that basically philanthropy and doing science, just like any other discipline, requires practice and you’re not going to be good at it overnight. So we should just kind of dig in and start doing a few different iterations on it and see what we enjoy and where we think we can have an impact and go from there. So, yeah, I mean, like you mentioned, I mean, this is we’re coming up on in November, the 10-year anniversary of when we started CZI. And, you know, there’s a lot... There’s a lot of work that we’re really proud of, that we’ve been a part of, including, you know, work in education and supporting communities. But, you know, when we reflect on it, we feel like the work that we’ve done in science really has had the biggest impact and in a lot of ways is accelerating. And with, especially with all the advances in AI that are coming, I think the ability to have an even bigger impact over the coming decade is... It just seems really clear like this is coming into focus. So, you know, for the next period, we really want to make science the main focus of what we’re doing and specifically the Biohub organization that we’re really proud of this model that we’ve helped pioneer that we can go into detail on is really going to be like the main focus of our philanthropy. And it’s just something that we’re very excited about.

Priscilla [00:02:17]: Yeah, when we started 10 years ago, we had this idea like, okay, I bring experience as a physician, Mark’s an engineer, and he builds things. And we have an opportunity to give back resources to... And we sort of just, we tried a bunch of things. And the thing that in running a philanthropy, I’m incredibly envious of people who run companies is that like you guys can have a dashboard and there’s like financial results and people tell you if you’re on the right track, on the wrong track, and there’s clarity. But in philanthropy, there’s so much you can do. And it takes a long time for you to get a sense of like what has momentum? What are we doing that is actually bringing all of our both skills and resources? So over the past 10 years, I would say we’ve been getting a sense of what is that thing that really allows us to have the impact and makes the most of what we bring to the table. And it’s really been around AI and biology where we’re like, oh my gosh, this is it. And, you know, the ecosystem is big. And we really think our ability to bring great scientists, great AI researchers together between the wet lab and the lab is really important. The compute, the ability to bring physicians and patients into the picture, that’s a unique niche for us at the Biohub. And that’s, you know, we need others to take the work to translation. The Gates Foundation has a strong focus on translation and the field. And we have had a number of really awesome collaborations and continue to where we really look at sort of the basic fundamental research. And being able to partner with someone who’s thinking about the translation layer is incredible.

Alessio [00:04:05]: We kind of see the first decade, and I would love to get your take as a decade of creating data, creating a science ecosystem, and then starting to work on some of the models. And the next decade maybe is more of the applied modeling side. At what point did you decide that just doing the tooling was matter versus, you know, you could have cured malaria in Africa too or some other disease, you know? Yeah.

Mark [00:04:31]: Yeah, I mean, take a step back. And this is kind of related to your first question too. I mean, there’s, like Priscilla was saying, the space is huge. You know, there are lots of other philanthropies, including Gates, who I think they would say that they’re primarily focused on public health and sort of administering. Like once you know what a cure is, just getting it out to the world is a huge thing too. And someone needs to do that, and that’s a lot of work and a lot of resources, and it’s good that they’re doing that. Basic science is another completely different part of the kind of innovation funnel to enable that. And our view is that the federal government basically dwarfs everyone else in terms of how much they invest through NIH. But there’s a certain pattern to how they invest, which is really enabling a lot of individual investigators to do work. And our kind of observation is that if you look at the history of science, a lot of major advances are basically preceded by new tools or new ways of observing things, right? So the initial telescope allowed. A lot of advances in astronomy, the microscope, the invention of that allowed a lot of understanding of biology. And similarly, I think we’re at a point in history where a lot of new tools are being built, computational tools, tools to instrument the body in different ways and understand things. And often those tool development just takes a longer term time frame and a sometimes a larger commitment of capital, including the way to do it. Isn’t necessarily just to make. A lot of different people. You need to really operate it yourself, which I think is one thing that’s different about the way that we’ve operated than than others is, you know, most times when you think about philanthropy, you think about kind of giving money away in terms of grants and a lot of what we’re doing is actually building up these institutes and kind of building labs to do that kind of research ourselves by bringing in leading scientists and engineers and all that. But that’s kind of the strategy is we feel like there’s a lot of new tools to develop. There’s sort of been a. A hole in the ecosystem where tool development and kind of the 10 to 15 year runway that you need to do that and often hundreds of millions of dollars to build things like the microscopes and that you’re in imaging that you’re seeing in this institute here. I think that that’s been sort of underfunded and that’s where we think that if we we do that kind of work, it can just give all these other scientists way more tools to accelerate the pace of of of research, hopefully discover cures. And then you have folks who are focused on public health who bring that out to the world and kind of deploy it to everyone.

Priscilla [00:07:00]: Yeah. I mean, our mission is to cure, prevent all diseases and like that’s not going to happen just in our four walls. So the strategy has to be how do we make every single scientist and everyone better and more effective? And, you know, the strategy Mark talked about is sort of where we landed on how to actually maximally move the field forward.

swyx [00:07:22]: Yeah. Yeah. The mission is cure, prevent all diseases. By the way, a lot of people outside of the CZI worlds are still kind of find this concept very alien. But talking to the CZI people, they really, truly believe it. And it’s impressive how you picked the right mission to motivate everyone to work towards this enormous task.

Mark [00:07:39]: Well, it’s kind of a funny thing. I mean, it’s I mean, I kind of we like to talk about the mission is like helping scientists do it, right? Because we’re not actually curing the diseases. We’re just trying to build the tools, advancing the data models. Yeah. Like basically accelerating scientists work towards that. But, you know, a funny thing about it is we had this initial time frame of by the end of the century. And, you know, if we. If you ask biologists, there’s a lot of questions are, OK, that’s really ambitious. Are we going to be able to do that? And then when you ask AI people, it’s like that should be really easy. Like, why are you so unambitious that you’re shooting for just the end of the century? And I do think that at the pace that that AI is improving things, I mean, I think it might be possible significantly sooner than that. I mean, I don’t think it’s necessarily worth putting a number on it or a date. But I think that that’s. It’s your point about the first decade was, you know, sort of about doing work like the Cell Atlas to to be able to help understand basically all of the kind of specifics and data about all the different configurations of every cell in the body. When we did that, we kind of had this vague notion that that would be useful to advance science. But I think that, you know, like a lot of people in the tech industry, we we have even been impressed by how quickly AI has accelerated. But that ends up being a really valuable thing to have done over the last 10 years. Especially for where AI is now and now the models that can get built with that.

Priscilla [00:09:04]: But the thing that’s interesting to agree is like, OK, so from a tech, I totally agree that in our intersection of AI and biology, the AI folks are like, yep, the biologists are like, hmm. And I think it’s actually that confluence of conversations that lead both the biologists to be like, OK, I’m really uncomfortable about this idea and timeline. But if I’m really pinned down to think about it. What are like you really force people to think through, like, OK, what are actually the barriers? What would you need to do? And you’re forcing that conversation from the biologist side and from the AI side, really getting a sense of, OK, what like data is not just data. You guys know this. Like you need to know sort of how the data was collected and from where and being able to connect the AI researchers to the folks who are actually gathering the data on a daily basis makes their work better. And so it’s it’s that conversation that’s happening here that I think makes people outside so excited about this because it’s credible. And they sort of have work really dug in and thought through how to how that would work. And and they’re excited and they believe and believing is the first step.

swyx [00:10:21]: There’s a general pattern of software eating the world. And I think AI eating the world is kind of like the next version of this. I was talking with Garrett outside who says he’s a biologist. But, you know, I think he’s using models. Like Sam from from Meta.

Priscilla [00:10:31]: You’re like, you don’t look like only a biologist.

Mark [00:10:34]: What does a biologist look like?

Priscilla [00:10:36]: I don’t know. I he’s working on models out there. And that’s like biologists are working using models, right? They’re not just like in imagination, like just using in the wet lab. Yeah, that’s totally.

swyx [00:10:46]: Yeah. I think one of those things that, you know, referencing the wet lab is all of the key approaches that you’re pursuing is turning things in, pursuing a virtual cell, turning things from mostly wet lab into something in silico. How far along are we?

Mark [00:11:00]: I mean, it’s pretty early, right? I mean, I think the first step, which I think is easy to overlook, is basically what Priscilla was talking about of just getting these folks together. It almost it’s worth taking a beat just to talk about this, just because I think most people assume that this is like, obviously, you would go do that. But it’s somewhat novel in science because of, I think, the way that a lot of funding has been done. That is, basically, you grant individual teams, like relatively small grants, and people do a lot of science independently. It is, I think, pretty amazing how much progress you can make if you just have people from different disciplines sit together, right? I mean, this is like over my career. I mean, both at Meta and here, it’s like you have teams that are not working together for some reason or they disagree on something. It’s like, okay, physically just have them next to each other. And it like actually is super helpful. So here, what are we doing? It’s not just bringing together the biologists and the engineers, which was a core part of the initial. Biohub model, but it was also unlocking the ability for people to work together across institutions. So the first Biohub that we started out here between Stanford, UCSF, and Berkeley allowed a lot more collaboration between scientists and engineers at those universities than was in practice happening before. And it’s like, you can look at this and be like, all right, that seems really obvious. But it actually was sort of an interesting and novel experiment and one that I’m really happy to see others also implementing. Because I think it’s really interesting. It’s just such a clear win, just the kind of the human side of bringing people together and having them sit together. So anyway, that I would say is kind of step one or step zero and is probably quite overlooked, but is sort of a fundamental part of the model that I guess also goes back to this idea of like, we’re not just kind of like granting funds to other people. We’re building an institution. We’re having people sit together. So then you get that and then you get these people who are like half biologists, half AI engineer, because they kind of have some experience doing it. And, I mean, I don’t know, I mean, we can we can talk through the specific models, and there’s a lot of exciting stuff there, but I’d say it’s, it’s an early glimpse of where this is all going, I think, like, you want to kind of build up these models, hierarchically, so you give them a lot of data about specific proteins, and they can model specific proteins in the cells, and then you can model different cell behavior, and then eventually you get, you kind of zoom out and you’re modeling like a virtual immune system or something like that. And it’s sort of hard to simulate the immune system without having a good understanding of how a cell might work. And it’s kind of hard to understand or simulate how a cell might work if you don’t really understand how the proteins interact. So you kind of need systems that understand data at all different levels of this, and then you kind of pull them together. And then if you look at the different models, there’s, you know, there are versions that are kind of focused on, all right, like which parts of the genome are kind of being expressed in different ways. I mean, the cryo. model that I think is very interesting, that’s built off of the data here, the only model that I’m aware of, that’s like a spatial model of like, of basically, like how, how these cells work, and you kind of, you just want to be able to look at stuff from different perspectives, and then put them together, and you build like a richer and richer model of kind of how these cells work, but we are definitely at the beginning of this journey.

Priscilla [00:14:19]: But it’s like slow and fast, slow and fast, right? So when we built the Human Cell Atlas, we started 10 years ago, it was one of our first RFAs. And we actually, the first RFA was was to fund the methodologies of how you would get a single cell transcriptome. And it took us about 10 years to get to a place where we had, we now have one of the largest corpus of RNA transcriptomes, 125 million cells, cost a lot of money. And the really cool thing we discovered through that process was, if we could seed the effort and make it easy for people to contribute, it happened. That’s cell by gene. We actually, we’re responsible for maybe 25% of the data, and the rest of the ecosystem contributed 75% of that. That’s an incredible asset and has been very important in modeling work. Similarly, if you look at AlphaFold, they built off publicly available data that was collected for 30 years prior, right? So that takes a long time. But now we’re doing the Billion Cell Project, and that is taking months, and at a fraction of the price. You know, really slow. To fast. But it’s a single dimension, and cells are so complicated. And here we’re looking, like Mark said, at the three-dimensional imaging structures. That’s, and it’s slow and expensive. But with cryo, with the cryo model, it will get fast again. And you just have to repeat it. And so I think we’ll get growth spurts, but it’s all happening just faster and faster.

Alessio [00:15:50]: How do you think about the layers? So you have compute, and we’ll talk about that later. On the data side, you build these amazing microscopes. I learned that they’re all built for you, by spec. They’re not off-the-shelf things that anybody can buy. They’re design partners. Yeah. How much of a bottleneck is that still? Like, can we convert the world of atoms into bits now at the right precision? Some acceptable rates. Or do we need more work on the microscopes themselves, too? I mean, you’re never done. Right. Yeah.

Priscilla [00:16:20]: Well, speed, for here, speed has been a big question of how, just getting the process through. So here we’ve worked on sort of the speed at which we can look at tomography. And then we’ve done the I don’t know the exact number. There are, like, maybe tens of these microscopes in the world. So that’s one bottleneck. And I think really is, like, what I was saying is slow and then fast. There’s so many other dimensions that we don’t have yet. Like, the cool thing is that we don’t have the space. thing here is with the transcriptome work, we’re looking at cellular expression. And with the imaging work, you’re able to localize it in space. And now you want to connect those two. But that’s still like two dimensions connected. Time is another dimension. We need to get dynamic imaging in place. Oh, God. That’s so much resolution. I know, right? But like really cool biological innovation. We need innovation in the way we can look at things like stain-free, dye-free, so we can look at things without sort of human intervention with time as a dimension is another. Because like, we are not frozen slices. So I think it’s just continuously looking at what the next dimension we want to sort of be able to either understand deeply or connect

Mark [00:17:45]: to our existing corpus of data and knowledge. And obviously, the ideal would be you want to increasingly be able to image things inside living cells, right? So I mean, you can kind of you can simulate it a bit by, okay, you can take a cell out, or some culture. It’s all destructive. Yeah. It’s like, okay, it’s living for a little bit or something. But I mean, you really want to be able to kind of as much as possible, actually understand what’s going on in living organisms. Can that be done? Is there what what what it approaches? Well, the better it gets.

Priscilla [00:18:14]: Well, there’s this cool methodology. So there is a really high intensity x-ray methodology you can use. The organ has to be dead. So like, you can just shoot x-rays, high intensity x-rays at like a lung and understand and like a sort of molecular level how the lung is assembled. And then you can correlate that with living imagery, right? MRIs of the lungs, CTs of the lungs, and look at the associations between the living images and real patients with the sample that you put into the high intensity x-ray. So that’s another example of like correlating data types, so that we can get that sort of high level specificity. With clinical data that impacts humans.

Mark [00:19:00]: But I mean, in some level, that’s sort of the point about building these AI biological models is you can have a lot of data, and you can interpolate that on that space and understand that. And then there’s, you know, so one of the models that, again, I mean, this is it’s really early work. And but the the R bio model, the idea of doing reasoning is that then you don’t just get correlation, but you get some understanding of like logic over how these things get together. And so that’s one of the things that we’re trying to do. And so that’s one of the two. So yeah, I mean, I think it’s probably going to be a while and people don’t have great hypotheses on how you’d actually do, like molecular imaging, like of a cell deep inside a living organism. But the goal is to be able to approximate that as much as possible with like this kind of surround view of, of different things that you can image.

Priscilla [00:19:48]: You guys like to see cool stuff. It’s not here. But at our San Francisco sky, we do image see through fish called zebra.

Mark [00:19:57]: That’s another, it’s another good example. It’s another good model. It’s like, all right, it’s like how, what’s a good way to imaging a living thing. It’s like, take a see-through thing.

Priscilla [00:20:06]: then use a model to say, how does this see-through thing actually relate to us? Right? Like, I’m like, not that interested in curing disease, cure, prevent, manage all disease for zebrafish. I am very interested. Mark’s pro zebrafish. I’m okay on zebrafish. But you, you need to use another application. Yeah, I mean, I think that’s a good point. I think that’s a really good point. And I think that’s is looking at how what is conserved and what is actually relevant and important to the way human biology works in a fish model. And so being able to have that translation be more effective, so we don’t waste our time on things that won’t apply in a model organism is another really interesting way to elevate biology.

Alessio [00:20:48]: On the data side, can you just give an overview of how far we are? Like what percentage of all cells are we image? And do we have, what’s the distribution? Yeah. What’s the distribution of them? You know, like when you say 150 million to 1 billion cells, is that a lot? Is that 10%?

Priscilla [00:21:05]: The funny thing is, until recently, we didn’t know how many cell types the human body had.

Mark [00:21:08]: Yeah, I mean, this is kind of a wild thing. I mean, this was a big part of the Cell Atlas project is like, there wasn’t even, it’s kind of like, imagine the periodic table in chemistry, but you, you know, it doesn’t end.

Priscilla [00:21:18]: You don’t have the squares. We know it’s billions. We know there are billions of cell types in a human. And we’ve only truly looked at a fraction. And we looked at it in largely healthy cells. And so, like, just the number of permutations of like, age, well, species, because not all research is in humans, right? So species, ancestries, like, what is your sort of genetic background, age, like, babies are different than old people, gender, all those things actually are permutations, environmental exposures, all those things are permutations on the cell that actually you want to be able to use. to understand in healthy and disease states. I feel confident that we are at the beginning of this.

swyx [00:22:03]: I’ll ask a little bit of an obvious question in terms of the intersection of AI and bio, which is, don’t we want precision in biology? Don’t we want some grounding in a world model, maybe, that we don’t normally get in a language model?

Mark [00:22:19]: Yeah, I mean, I think that that’s sort of the point of doing all the measurement and being able to have all this real. Yeah, I mean, it’s like, so you have the... The diffusion model for generating cells that we put out. And it’s like one of the recent models. And it’s like, it’s cool, because you can basically, you have a model now that you can describe is like the conditions, and it’ll basically give you a synthetic cell. But yeah, you want it to be increasingly grounded. And that’s a lot of the point of the biology and the engineering that we’re doing is to be able to have these different facets of that. So the imaging institute is one part that gets you the spatial data, that’s, that’s very helpful. And the work that we’re doing in the other bio hubs on cellular engineering and instrumenting inflammation and things like that, it’s basically, it’s scientific work to build new types of tools that allow us to measure new types of things that generate data that allow us to ground the models in different ways. One framing that we have on this, that I think is, is pretty interesting, is that, you know, there’s this sort of... There’s this concept of a frontier AI lab that is like, okay, it’s, it’s building AI models that are sort of at the frontier of what’s possible. And I think you can think about biology in that way, too. And there’s sort of a concept of a frontier biology lab. Like, what is the idea of a, you know, it’s like labs that are kind of at the cutting edge of, like, building the most advanced imaging, like measuring, you know, inflammation or doing cellular engineering in the most advanced ways, whatever the problem space is that you’re at. And then I think that there’s this interesting problem space of what happens if you’re at the front end of the lab. At the intersection of those two areas, right? So, I mean, you mentioned the work that DeepMind did on AlphaFold, which is great. That’s an example of a frontier AI lab using a data set that was just generated by other scientists, like over decades, right? But I think part of what we’re trying to unlock here with Biohub is the idea of what actually, what happens if you do frontier biology and frontier AI in sync together, and you’re designing the tools on the frontier AI lab. I think part of what we’re trying to unlock here is that you’re designing the tools on the frontier AI lab.

Alessio [00:24:56]: What’s the hill climbing in this scenario? So, like, with language models, you have benchmarks, you look at the benchmark, you just make that go better. With these things, you have to bring it back to the real world. So, as you build these models, like, how do you bring the two teams together to get feedback?

Priscilla [00:25:11]: think it’s very similar to what Mark just said. You want to be able to validate on the accuracy question. We don’t expect that these models, they will get increasingly accurate. But you want to be able to have feedback. And it’s not as easy as being like, you know, this, yeah, this output doesn’t make sense. You have to actually take it to the wet lab, run the experiment, find out if it actually happened as predicted, and feed it back into the model. And that’s the virtuous cycle we want to build to help the AI best serve the biologists, and the biologists be part of continuously improving the models.

Alessio [00:25:47]: From like a numbers perspective, in a language model, you can run tens of thousands of tests. Very false. Yeah. And we have to build a lot of them out. Yeah. Yeah. And then on going to the wet lab, what do you think that’s going to be like the feedback cycle? Like, as you start to have more of these things to be tested in the wet lab, do you feel like that’s going to be a bottleneck that like we cannot take that many?

Priscilla [00:26:09]: I don’t know the answer to that yet. I think the throughput on sort of established metrics in the wet lab is actually getting quite fast. You can run, parallelize a lot of experimentation. So, but it’s not at the, easily at the tens of thousands of verifications, but it’s, we’ll have to, well, we actually have to see, we’ll probably need to be smart about how we do it.

Mark [00:26:35]: But I mean, there’s, you know, a lot of people I think often take these things to the extreme and are like, okay, pretty soon if you have these models, you’re just going to be able to run experiments with the models without even having to go to a wet lab. And it’s like, no, I mean, I think that’s kind of like, I think that that’s sort of the biological version of like, eventually, you’re going to be able to run experiments with the models without even having to go to a wet lab. And it’s like, okay, maybe you get there, right? And I think that there’s like some chance over time, but well, before you do, you’re going to be able to have models that can help generate hypotheses and scientists can apply their taste on which ideas or kind of suggestions come from this are worth testing. And then you test them and then you feed it back into the model, which I think is basically the way that every AI model is deployed into work in other places. Totally.

Priscilla [00:27:24]: Like you right now, because the wet lab is so expensive and relatively slow compared to sort of computational experimentation, like people are choosing, like, I need something to hit. So people are going for hypotheses or ideas that are like, you know, to use a sports analogy, like singles or doubles. But like, it’s just too risky. They only have so much grant funding and they need something to help move their work along. But like, if we have a model that can help de-risk some of the bigger, riskier ideas, that’s going to move science faster. And I think makes the science and those ideas both, you know, can be sourced with AI as a tool. But really, it’s really about making the scientists less hesitant to explore big ideas.

swyx [00:28:13]: Yeah, obviously, that’s a lot of the success of the model of CZI, which is serving this part of research that is underserved. Because there was basically no benefactor or no funding mechanism by which to do this. One thing that we’re announcing when we release this podcast is this unification of the sort of biohub model. I think it’s very analogous to the foundation model in a frontier lab approach, right? Where you bring together people in different disciplines, you have much longer time horizons than anyone else. Are there any other key elements to the strategy of the biohub that you’re taking?

Mark [00:28:46]: Well, I mean, one thing that we haven’t talked about is the evolutionary scale team. And Alex, Well, let’s talk about the announcement. Yeah, so I mean, I mean, this is like, probably the most talented team working on AI and biology, right? And like at the intersection of doing of like, basically, good biology background. And also, you know, they’ve just been working on ESM3. Yes, and the top protein models for a long period of time. Yeah, I mean, I think if you want to build a, an organization that is doing frontier biology and frontier AI, you need to have like world leading. Yeah, I mean, I think if you want to build a, an organization that is doing frontier biology and frontier AI, you need to have like world leading. researchers and we’re doing that by basically combining the team that we have that’s already put out all the models that we’re talking about today, plus having the evolutionary scale team, which is just like, very renowned. Join and Alex is basically going to be running the program. So I think it’s, it’s sort of an interesting decision, I think, to have the AI person basically be running the overall program, partnering with these leading, leading biologists, I think gives a sense of how optimistic we are about the AI work being very fundamental to this. But we’re very serious about building out like a leading part, a leading lab on the AI side as well. That goes for both the talent and the compute. I think we were probably the first to build out a large scale compute cluster for, for biologics. I think now there’s some others who are doing it too, but we’re also building on that and, you know, we really, we plan to release frontier models on this.

Alessio [00:30:15]: Do you see that as the 10 year output, like in the next 10 years, we look back at that day. I just talked to them yesterday.

Priscilla [00:30:21]: They say it’s faster than that. But AI people are always in a hurry.

Alessio [00:30:26]: We have AGI in two years. So would that be a satisfactory result for you guys? You fast forward 10 years, you have like the best, you know, the three best models in, in biology, or is there like a further development? What’s your goal that you want to have as an output of the foundation?

Priscilla [00:30:40]: I have to bring it back to the patient. I think like the AI models are, I think we’ll be very excited, both if we have great models and scientists are using them, but you really want to make sure that it’s like accelerating clinical impact. Like that’s the goal, right? Like that the AI models is a very challenging milestone that we’ve worked, we are working very hard on and we will get there. But how do you actually take those models and apply them to actually change the way? People live. And there’s so, there’s two variants that I think about in the application of these models. Why are they important? One is like each one of our genetics is incredibly diverse and different. First, like, first of all, we are just all, the four of us are unique people, but we also have things like that are sort of known indicators of disease and unknown indicators of disease. And I actually find the variance of unknown significance to be the most important. The most interesting and the most frustrating. Say, you know, someone that you love, it’s sort of a diagnostic mystery. They need to go in and look at the genetics. Most likely they’ll come back and be like, there are these three things that are not usual, but we also don’t know why. And you’re like, okay, like, should I panic? Should I not panic? Like, what do I do now? And what you really want to do, and I think these models will be able to do, is look at those variants and actually model out what is the impact in the different cells. How it influences the cells. How it influences cellular behavior and whether or not that is tied to a pathway to disease or not. Like, that’s a big deal, and I think we should be doing that. That is actually the future of medicine, where we think about each one of your biology based on your genetics, your exposure, and how it predisposes you or not to disease. Like, that’s huge, and we want to be able to see that clinical application, but we can’t. It’s too expensive, too hard to model each person. It’s impossible to model each person in the lab, but if we can build models around this, it is possible. And then we can start thinking with extreme precision. And I’m not just talking about rare disease. Like, there are, like, common diseases. I’ll just say depression. Right now, it’s empirical, right? We just say, like, you’re depressed. Like, here, let’s try this antidepressant. And it’s, like, usually the one that the doctor is more familiar with or maybe one that you’ve heard of. But, like, and then you have to try it for months. Before it’s, like, did it work? Did it not work?

swyx [00:33:09]: Months? Yes. That’s the cycle? I don’t have familiarity with this. That is the cycle. That’s horrible.

Priscilla [00:33:13]: And meanwhile, if it doesn’t work, it means the person’s suffering. And this applies to, like, almost every disease, right? There has to be some biological explanation as to why some medications work and don’t. So can we actually then look at each patient and say, based on who you are, we think this medication is going to work best for you? That’s the future I want to live in. Where we can actually understand individuals as individuals and use the biology and science very directly to keep them well.

swyx [00:33:44]: Yeah. So, like, if there’s a name for this tool that has the clinical impact that is on the scale of the electron, how do you envision it, I guess? Like, I feel like it’s almost going to be the CZI app, I guess.

Priscilla [00:33:59]: Oh, well, it won’t be. First of all, that’s not what we’re building right now. We’re building the basics. We’re understanding, like, cells and molecules. Someone else will do it. We’re painting a picture, like, we need partnerships. You asked about the ecosystem before. Yeah. Like, there are experts along the way of this pathway. And so we sort of are at the fundamental research side. Yeah. And you need to be able to partner with folks to bring this all the way through impact. But the way I think about it, people call it different things. But essentially, you want to get to medicine where it’s truly precision medicine. Yeah. It’s N of one. We’re understanding you and designing therapeutics for you.

swyx [00:34:39]: Yeah. I like the mission of rare as one as well. That’s a great framing.

Alessio [00:34:43]: Do you feel like that’s possible? Like, almost treating the body as, like, a compiler? It’s like, because I know exactly what it looks like, I know exactly what’s going to happen?

Mark [00:34:51]: Or is the body just, like, there’s too many outside inputs and, like, over time it kind of deviates from what you have? Well, I think we’ll see how far we can get. But I mean, I’m pretty optimistic that we’ll be able to make a bunch of progress. And yeah. I mean, there’s, like, you basically, I mean, what format does this take technologically? I would imagine you’re taking these different types of virtual cell models and eventually merging them into the equivalent of, like, a biological omni model. Kind of like how on the language model side you had people that did language and then, you know, people who did different kinds of media models and perception and all that. And then eventually you just kind of merge that. And then, you know, you aim to get positive transfer by merging it. So that way it’s not just. Just combining capabilities, but getting everything else to be stronger. So, yeah. I mean, technologically, I think that’s basically what it looks like is, you know, over whatever it is, a five or ten year period. We’re building up a series of biohub models that, like, increasingly get all these different dimensions of data and capabilities that can be used to help run individual science experiments and potentially eventually help with finding individual therapies. For patients, although we’re going to be less on the clinical side, we’re going to be more on the kind of scientific tool development side. And the kind of main tool, if you will, is this, like, these biohub virtual cell models.

Priscilla [00:36:16]: I would say five years ago, without sort of the large language model supporting this, I don’t think it would have been possible to really because biology is incredibly complex. And what we’re essentially trying to do is break it down from a discovery based science where we’re trying to do a big discovery based science. You kind of get lucky, you kind of get clever, and you sort of figure out a hack to learn something new to really making it closer to an engineering problem of like, this is how the system works. And when this breaks, what happens to the rest of the system? But like you said, there’s just there’s far too many dimensions for us to hold in our brains. That’s why we’re so excited about this intersection at this moment, because it is possible to consider so many more dimensions matching the complexity of biology.

Alessio [00:37:03]: What is the role of the doctor in the future? Right? If you can like predict everything out, and then if you take personal super intelligence seriously, do you kind of distribute some of the diagnosis and all of that work? Or how do you envision that?

Priscilla [00:37:18]: I’ve been thinking about this a lot. And I think one is, the model is not going to take you all the way, you’re still going to need to really look at individual clinical situations. And, and the doctor is going to be a form of data input into the model, right? And so the doctor, there’s some judgment that comes into place. But there’s already a lot of models that make doctors really good at what they do. For instance, looking at your skin, like, AI is really, really good at detecting lesions in your skin that are concerning. It’s excellent. Retinal issues, it is excellent. So the AI modeling and mapping is really, really good. So it’s already happening. So I think about like, what should future doctors be trained to do? And I really think care and compassion and sort of walking patients through understanding, I think understanding why leads to trust in both the science and in the clinical pathway. And really walking alongside patients on that journey is, you know, it was the original calling of physicians to be healers and to be using great tools to heal patients.

Mark [00:38:33]: Well, so bedside manner, ultimately, is, I mean, I also think you can zoom out, though, from like the role of a doctor to I think everyone wants the health system to be more proactive and less reactive, right? So today, it’s like you show up when you’re sick. And then like you have someone treat you or understand what’s going on. I think the goal with a lot of these systems is to be much more proactive about this. So when we say that the vision is to try to help scientists cure and prevent all diseases, it doesn’t mean that there’s going to be like no bacteria in the world. And like, no one ever starts to get an infection. It’s just that, all right, ideally, you can kind of understand all that really early, right? Similarly, if someone, you know, if someone gets a mutation, it looks like it might become cancerous, then you can just treat it a lot better if you if you know that early, rather than like showing up to a doctor when it’s, you know, already metastasized, and you have a bunch of issues on that. So, so I don’t know, I think that there are gonna be a lot of opportunities to fundamentally improve the healthcare system overall. But I think, I mean, I agree with everything that you said on this. And I also just think that, like, when we say that, that we think it’s gonna be possible to prevent and cure all diseases, it’s not literally that no one ever gets the beginning of a sickness. It’s just that it like, it kind of can be managed in a way where everything is sort of manageable.

swyx [00:39:56]: I think we discover more diseases, the longer we live. Is it possible to not die? You know, obviously, that’s a that’s a meme that’s coming to fruition. If you theoretically cure all diseases, maybe death is a disease.

Priscilla [00:40:11]: Mark just said we had extreme alignment, which I love. Thank you, honey. This is this is one that we don’t know.

swyx [00:40:18]: This is one that I’m not sure we have extreme alignment on. I in fact, just haven’t thought about this one very much. Because I think there is so much there’s other things to do.

Priscilla [00:40:27]: There’s so much to do in terms of like, you know, I’m a pediatrician. I think about babies, and like, very sad things happen to very small people. And like, I think a lot about that. And how do we like maximize life quality and the things that harm small people? I’m biased. And I haven’t thought as much on the other end of the spectrum. But I don’t know, I’m 40. Maybe I should, but I feel like I can still focus on the little ones.

Mark [00:40:55]: I think the strategy is the same, right? I mean, it’s like, we’re, we’re basically choosing to not focus on any specific disease and like verticalize. Our strategy is one of trying to accelerate scientific progress overall. And I think that there are a lot of people who are going to focus on each of these individual things. So I don’t know.

Priscilla [00:41:13]: But we don’t have to because that’s not our strategy. Our strategy is to make sure that we have tools that make people do the best science possible out there.

swyx [00:41:21]: I’ll put to you that, you know, because of because aging and environments. And mutations are so diverse, like you actually, you have a high concentration of grouping in the early years, and it should have more diversity in terms of the cell types and the problems that you face in the later years. And so that I mean, there might be some imbalance in terms of where all these things happen. But I’m not, you know, I’m not I’m not pitching it any particular direction.

Mark [00:41:47]: I mean, I think it’s clearly if you look at the trend over the last, I don’t know what it is 100 years, I mean, there was this flip. And if you like, pay attention to the history of science, where it changed to kind of hypothesis driven scientific method of like, we’re going to run tests and have controlled experiments. And since that happened, the average life expectancy is basically increased by I think it’s about a quarter of a year every year over the last 100 years. Now, a lot of that, like Priscilla said, is basically making it so that a lot of people don’t die young. Yeah.

Mark [00:42:25]: maximum human life expectancy, although the oldest people today, I do think in general are older than the oldest people, you know, 20 or 30 or 40 years ago. But I think that there’s been a little bit less of an increase there, and more just kind of making it so that people don’t suffer and die prematurely from things. But I mean, there’s other things that you want to focus on here, too. It’s not just like, how long you live, it’s like health quality of the life while you’re, you know, so I think it’s like, you can live, you can die, you can die prematurely.

swyx [00:42:55]: You can live a full life and have that be high quality, you know, or you can get sick in different ways that kind of add up over time. And I think like, there’s lots of different ways to improve. I mean, it’s, it’s, there’s like, you know, all these different analogies that you can throw at this, but I think there’s, there’s just a lot of, a lot of room to improve here. Yeah. And then the other element I wanted to come back to on the engineering side, which is when you presented a high dimensionality problem, you want to reduce things into little boxes that you can sort of manipulate at a higher abstraction. And that’s something I try to do with the folks outside. And it’s really, we really struggled because over here, you’re imaging on like the atomic level. And then you’re also worrying about proteins. And then you’re also trying to build a cell model. Is every abstraction leaky? Like, where’s the boxes I can move around and not worry about it? My physics analogy is in the regular world, you don’t have to worry about quantum physics. But here we kind of do.

Mark [00:43:48]: I think you want to build it up a little bit hierarchically. And like, and when you’re trying to understand proteins, understanding molecules. Makes a big difference. But at some level, you can kind of just look at like correlations and cells. But if you want to like really have the most accurate model, and if you want to be able to reason about things, then you probably also want to understand proteins well. And then I think that kind of extends. But yeah, I mean, that’s part of the interesting challenge of this is that it’s not just like one, one resolution that you’re looking at it. I think in order to do it well, it’s, yeah, I mean, you’re, you’re kind of, you have some amount of abstraction, but I think you want the models just like, you know, like, you know, like, you know, like, you know, like, you know, like, you know, like, you know, like, you know, like, you know, like, you know, like, you know, like language models, or I think how our brains work to basically build up different levels of abstraction and pattern matching. And I think that that’s here too. And you basically just need to be kind of like have some basic excellence and understanding at each of these different levels.

swyx [00:44:37]: It’s weird that the number of levels in which you have to telescope up and down, it’s mind boggling. And I think like, that’s when people say dimensions, they typically mean orthogonal dimensions. But here is sort of like nested.

Mark [00:44:49]: Yeah, yeah, yeah, just different scales that are like, oddly different disciplines.

swyx [00:44:55]: To understand each specific scale. And it’s like, in a way that, like, the people who are good at understanding one scale, are like never spoken. Yes, people at the next scale. Yeah, yeah, yeah, physics is there, chemistry is here. Yeah, yeah, bio is there. It’s nice to hear about it. But when you see it, and you meet the people, you’re like, Oh, this is real, like, and they are, they are actually working together. Yeah.

Alessio [00:45:16]: And then there’s this goal of the virtual immune system that you’re working towards. I would love to, for you to chat about that. And also, like, if that happens, what should other people build? So there’s obviously, you know, CRISPR, and some of that technologies, like the people should maybe ramp throughput for, like, how do you think about the future?

Priscilla [00:45:34]: The virtual immune system, I think, is, is, you know, obviously, I think of a subset of sort of the generalized model eventually we’ll get to. But the virtual immune system is super interesting for a couple of reasons. One, it’s individual cells interacting with each other. There’s, you know, a number of cells.

Priscilla [00:45:59]: And so we can use our current technologies to understand these cells at a more granular level. So that’s cool, from a biology standpoint. But the clinical impact is huge of understanding the immune system. Because biology, turns out, has already given us a way to keep the body healthy. And it also sometimes goes awry and causes disease with autoimmune disease, right? Yeah. It’s a very complex system that has to stay in balance. And if it goes out of balance in either direction, you get sick. It can also go into your body, and it’s a privileged system that is mobile and can go into places like your brain, your pancreas, your heart, to sort of either do maintenance or to collect signal. That’s, it’s built in. So if we can understand this system, we can use it to keep people healthy. We already kind of do. There’s CAR T-cells, where we reprogram T-cells to go in and fight cancer. And our New York Biohub, we’re doing cellular engineering to say, like, hey, can you go in to this person’s heart, check if they have plaques that are causing problems, read it into your DNA, self-lyse, and then we can read out the signal as cell-free DNA and give us a binary answer, yes or no. Right.

Priscilla [00:47:25]: That is incredible. That is a tool that, like, is realistic, too. I know it sounds sci-fi. It is realistic. It is happening. And then on the other end of understanding the balance, like, so many autoimmune diseases, MS, lupus, those are the ones, examples of ones we know. I think there are other things that are autoimmune that we don’t understand, like dementia can have a, autoimmunity can play a large role in that. And so if we can understand the fine balance that the system needs to be kept in, then we can understand the balance. Then we can actually impact a lot of the ways the human body is maintained. So I think it’s both interesting from a biology perspective and feasible to model, and probably one of the highest sort of now impact systems if we can learn how to manipulate.

Mark [00:48:11]: Amazing. But it’s only one system, right? I mean, I think it’s like the—

Priscilla [00:48:14]: So it’s a subset.

Mark [00:48:15]: If you’re focused on curing and preventing diseases, the immune system is a pretty important one. And I think it’s also interesting for all the—and unique in a lot of the ways that you said. But there’s, like, lots of other parts of the body to understand, too.

Alessio [00:48:27]: And I think we’re running out of time, so we have two questions to close. One, again, 100 years, maybe it’s too long, right? What would it take to do it in 50, in 25, and to make those happen, like, what should other people build to support your work?

Mark [00:48:42]: I mean, I think a lot of this is going to end up coming down to how far a lot of these AI methods get, right? I think that there’s, like, people have—there’s just this constant ongoing debate around, what are the timeframes for getting to very strong AI? And I think if you get that, then I think it’s pretty optimistic that with the right investments in frontier biology, you should be able to get these systems that can allow you to have virtual cells, that allow you to do the kind of precision treatments and preventative care that can achieve this kind of mission significantly sooner. But at the end of the day, I think a lot of that time— I think a lot of that time frame will probably come down to the AI time frame. There’s obviously a ton of stuff to do in biology, but it’s—so it’s not—I mean, like, I think that what should other people do? I mean, other people doing more frontier biology and helping to collect this type of data and solve these problems is super helpful to that, too. It doesn’t automatically happen. But I guess if we’re, you know, predicting whether it’s going to take 10 or 20 or 40 years, that is probably more a function of the pace of AI development than it is a pace of the pure biology side.

Priscilla [00:49:52]: Yeah, I was going to agree with you. I think it’s a—a lot needs to—it’s—I think we’re on a path to get a lot of important biological data through advances in laboratory technique. But it’s not a given. Like, and there are different groups that are expert at this all across the nation and across the world. And so we need to be continuing to push the research and the methodologies. And I want to say that, like, you know, the Cell Atlas was not glamorous work. People were not going to get— They were not going to get their 10-year track paper by sort of analyzing the 100th and 20 millionth cell. That is just not it, right? And so rethinking the way that this work gets done in a collaborative—like, doing big things together in science. Like, that’s what is going to need to happen to sort of get the knowledge we need to build models that give us this type of insight.

Mark [00:50:46]: I guess one thought on, like, the type of biology that I think should get done. Is there is a certain orientation around choosing problems that will help generate data that can help make the models a lot smarter. I think that there’s a—you do that when you are very optimistic about the pace of progress and what AI is going to enable. Because the classic reason that scientists generated data sets is so that they could basically look through the data sets to make advances. So it is a little bit of an inversion in the thinking, which is like, I’m now going to do this so I can, like, help train this other thing to be better and create more advances. And I think in a world where you really believe that there’s going to be very significant AI progress, I think more frontier biology should be done in that way. But these data sets aren’t going to get created by themselves. There’s a lot of work that needs to get done and a lot of investment there. And at some level, you could probably have the smartest AI model in the world. But if it doesn’t actually have the data to understand this stuff, it’s like, okay, you can’t just, like, reason from first principles about all these things. I mean, a lot of human knowledge comes empirically, not from first principles reasoning. I think that more—this is kind of the whole Biohub network idea that we’re building. And I’ve been really happy to see other folks, especially a lot of people in technology, I think, have this orientation, too. They believe a lot in AI. They believe in the technological progress. They’ve generated some significant wealth for building their companies, and now they’re investing in science research. And I think that’s great. And I think doing it in this way where you’re, like, building up these networks to solve specific—to basically build specific tools that generate data that make the models better, it’s one approach. It’s not like—it’s not that all science should go in that direction, but it’s one of the things that I’m quite optimistic about that I think is going to make a very big difference. Cool.

swyx [00:52:42]: That’s probably all the time we have, but I’ll just leave it to you guys for any calls to action, anything that you want biologists or engineers to check out. I mean, check it out. Check out the models. Check out the models, the tooling. Yeah.

Mark [00:52:54]: I mean, they’re early, but I think it’s kind of an interesting sense of where things are going. And, you know, we’d love feedback on it, and it’ll kind of just help this feedback loop of, like, what we should build next.

Priscilla [00:53:08]: Yeah. I would say let’s do this together. Like, we need lots of people coming together to do this work.

swyx [00:53:14]: Well, thank you for organizing it and solving and curing all diseases. Trying to help others do it. Yes. All right. Thank you, guys.

Priscilla [00:53:22]: Thank you.

One of the most potentially gamechanging announcements of the past month was OpenAI and Stripe’s collaboration to add Instant Checkout to ChatGPT:

There have been many attempts at monetizing agents (x402) and working out agent to agent protocols, including over MCP, but the entry of two category leaders in AI and payments is obviously impactful and needs to be watched closely.

Emily Glassberg Sands, Stripe’s Head of Data & AI, joined us today to talk about Stripe’s new mission to build the economic infrastructure for AI, and how they both use AI internally and see it as an advantage for the next phase of economic growth.

ACP is Stripe’s protocol jointly developed with OpenAI, which powers ChatGPT Commerce; it lets agents discover and purchase from merchant catalogs across ecosystems. The core components:

• Catalog schema. Products, inventory, pricing, brand constraints, etc.

• Shared payment token (more below)

• Risk signal. “Good bot/bad bot” scores ride with the token so merchants can decision intelligently.

The ACP is being coupled with Stripe’s Link product, which has 200m+ consumers, as the wallet for your credentials. (Fun fact: 58% of Lovable revenue comes from Link transactions)

SharedPaymentToken (SPT) design:

• Scoped to specific business, time-limited, amount-capped, revocable. Buyers use existing saved payment methods without re-entry

• Underlying credentials are never exposed to agent and it’s designed for compatibility with card network agentic tokens like Mastercard’s

• When processed via Stripe, includes Radar risk signals (fraud likelihood, card testing indicators, stolen card signals, etc)

Emily has talked about this elsewhere, but it needed to be covered.

ML for fraud detection was one of the “old” AI markets, but it was not super reliable. Stripe only had 59% accuracy on card testing transactions for example. Move from task‑specific models to a transformer-based domain foundation model that produces dense payments embeddings to power many downstream tasks:

• Treat each charge as a token and behavior sequences as the context window.

• Ingest tens of billions of transactions (50,000/min) with full feature breadth: IPs, BINs, amounts, methods, geography, device, merchant traits, and more.

• Focus on “last‑K” relevant events across heterogeneous actors and surfaces (login, checkout attempts, retries, issuer responses).

It lifted large‑merchant card‑testing detection from 59% to 97% and also helps merchant with transaction clustering, checkout optimization to understand user flows, etc. Every single Stripe transaction goes through this model in <100ms.

AI has added new types of fraud too as the cost of inference is now meaningful. A free trial abuse with a virtual credit card in the past would be annoying, but the cost of serving that user was low. Fraudsters are now burning tens of thousands of dollars of AI tokens using virtual credit cards or through chargebacks, which makes this much more important to get right.

Everyone is a big fan of the pareto frontier charting for model capabilities, but it’s extremely hard as a product builder to offer the user choice while having a static plan. At the same time, users don’t want to use dumber models just to fit your pricing and they’ll often want to pay more for more intelligence. Stripe built Token Billing which calculates billing based on the model:

The other approach that companies are taking is outcome based pricing, which we also covered in our Dharmesh Shah episode. Intercom charging $1 per solved customer support ticket is an example.

• Toolshed (MCP server). Central tool layer wired into Slack, Drive, Git, data catalog (Hubble), query engines, and more. Agents can both retrieve and act using tools.

• Text‑to‑SQL assistants.

  • External: Sigma Assistant works well because revenue data is canonical.

  • Internal: “Hubert” (used by ~900 people/week) sits on Hubble for broad data; biggest failure mode is data discovery. Most of their work has been on deprecating low‑quality tables, creating human‑owned docs for canonical datasets, and persona context (“which org you’re in”) to give as system prompts to bias table selection.

• Semantic events + real‑time canonicals. Re‑architecting payments and usage billing pipelines so the same near‑real‑time feed powers Dashboard, Sigma, and data exports (e.g., BigQuery) with one source of truth that LLMs can consume.

• Engineer adoption. ~8.5k Stripes/day use LLM tools; 65–70% use AI coding assistants. “Impact” measured beyond LOC/PR counts and more empirically on developer perceived productivity.

• LLM‑built integrations. A pan‑EU local payment method integration went from ~2 months → ~2 weeks, trending to days.

We figured we’d get some #s to fuel the “AI startup bubble” discourse on X :)

• Go‑to‑market velocity. Top 100 AI companies on Stripe hit $1M/$10M/$30M ARR 2–3× faster than a comparable SaaS cohort five years prior.

• Global from day one. ~2× the country count by year end; median AI company operates in 55 countries in year one, >100 in year two.

• Retention dynamics. Per‑company retention slightly lower than SaaS, but churn is often switching, not abandoning the problem; customers boomerang as products leapfrog.

• Tiny teams, big revenue. Revenue per employee outpaces elite public benchmarks. Unit economics depend on inference costs trending down; plan for volatility rather than current spot prices.

• 7x growth rate. Companies on Stripe grew 7x faster than S&P 500 last year.

On YouTube now!

• 00:00:00 — Introduction and Emily’s Role at Stripe

• 00:09:55 — AI Business Models and Fraud Challenges

• 00:13:49 — Extending Radar for AI Economy

• 00:16:42 — Payment Innovation: Token Billing and Stablecoins

• 00:23:09 — Agentic Commerce Protocol Launch

• 00:29:40 — Good Bots vs Bad Bots in AI

• 00:40:31 — Designing the Agents Commerce Protocol

• 00:49:32 — Internal AI Adoption at Stripe

• 01:04:53 — Data Discovery and Text-to-SQL Challenges

• 01:21:00 — AI Economy Analysis: Bubble or Boom?

• Radar (Stripe’s fraud detection system)

• Stripe Link

• Agentic Commerce Protocol (ACP)

  • ChatGPT Commerce

• Sigma

• Token Billing

Alessio [00:00:00]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, founder of Kernel Labs, and I’m joined by Swyx, editor of Latent Space.

Swyx [00:00:03]: Hello, hello. We’re back in the studio with Emily Sands from Stripe. Welcome.

Emily [00:00:13]: Thank you.

Swyx [00:00:14]: So Emily, you’re head of data and AI at Stripe. That’s a big title. What does that actually mean in practice?

Emily [00:00:21]: So Stripe is building financial infrastructure for the internet. We started out as payments infrastructure, and now we are helping businesses solve a whole range of problems. How do they accept recurring payments like subscriptions? How do they do usage billing, revenue recognition, tax, money movements, accept stable coins, and more? And when you think about what we’re looking at, we’re looking at on the order of 1.3% of global GDP. About $1.4 trillion a year is processed on Stripe. And so that obviously creates a very unique opportunity to use that data to understand both what’s happening in the economy. What do our users need? But also feed it back into the product to power better payments experiences, so cut down on fraud, drive the right authorization, better customer-facing experiences, optimizing the checkout suite, and more. So anyway, our data and AI org is just really focused on helping Stripe make effective use of our data. And that starts sort of all the way at the foundation layers, right? Like, what’s the data platform? How do we do data engineering? What are our ML infrastructure, AI infrastructure? And then all the way up to the technology. And then all the way up to the applied layer. We also have a fun little group. It’s actually quite small. It’s just two dozen people, but we call it the Experimental Projects Team. And it’s not data-specific, but the premise is experimentation can and should and does happen everywhere. But there are often these sort of cross-Stripe opportunities that are being pulled out of us by our users, just given the pace at which the world is changing, that aren’t natural or easy to jump on within any one product vertical. And so these are just quite senior, quite seasoned engineers who run at those opportunities and zero to one them and get them off the ground. So our agentic commerce work came out of that. Token billing, which we can talk about in a bit, also came out of that. And that’s just a fun sort of side angle within our group that’s proven very high leverage.

Swyx [00:02:18]: Yeah, I like the framing that Stripe’s mission is build financial infrastructure for the Internet. And your subset of that is build economic infrastructure for AI. Yeah. And that’s a pretty...

Alessio [00:02:30]: Ambitious goal. You’ve been at Stripe four years. At what point did AI become a title-level thing? Because, I mean, you were obviously using machine learning for fraud detection and everything. You were at Coursera. Yeah. Yeah.

Emily [00:02:43]: We started investing in AI or LLM-specific experiences basically when GPT-3.5 hit the scene. We were like, okay, we need everybody to be able to have high-quality, safe, easy... ...access to LLMs, not just for their own, you know, day-to-day work usage, but actually to build, you know, production-grade experiences. So that’s sort of, what was that, early, like, January 2023 or late 2022. We started reasoning about, okay, like, you know, it’s not just ML infrastructure. It’s also AI infrastructure. It’s not just ML applications. It’s also AI applications. But then it was really only in the last year and a half or so that we said, hey... I mean, we had, like, transformers and whatever before. But only in the last year and a half or so that we were like, hey, we actually need to have our own domain-specific foundation model. And actually, we can move from these, you know, single-task, point-solution ML models to, you know, a much richer, denser payments embeddings that can then power the various downstream applications. So I think it was an evolution for us. And, you know, I think we could still debate, like, what’s ML and what’s AI in the industry at large. But you’re right. We’re more than a decade into using ML at Stripe, you know, way back in the early days for not just radar, which I think people know about, right? Like, the machine learning systems that block fraud for our customers, but also ML internally for our own operations. Like, every other payment service provider is onboarding max a dozen users a day. We’re onboarding thousands of users a day. And you have to make sure they are supportable and not fraudulent. And are creditworthy because you are processing their transactions. And that alone requires machine learning and long has. Yeah.

Swyx [00:04:37]: I would say, you know, it’s kind of interesting how the domain-specific models first come up, like, pre-foundation models. And then we have this foundation model era. But then at the scale of Stripe, I imagine that you also have to just serve in so much volume of inference that then you might have to domain-specialize them again. Yeah, so there’s fine-tunes on top.

Emily [00:04:59]: There’s fine-tunes on top. I mean, we see, like, 50,000 transactions a minute.

Swyx [00:05:05]: But not all of that goes through your foundation model.

Emily [00:05:08]: Yes, every single transaction. So, for example, the foundation model, one of the things it powers is detecting card-testing attacks. Do you guys know what card-testing is? Yeah. Yeah, okay. So, for listeners, it could be, like, a card tester could be enumerating through cards or they could be random-guessing cards. They find a card that works and then sometimes they use it for fraud. More often, they sell it. Lots of traditional machines. Machine learning models can do a pretty good job detecting card-testing. But card-testers have gotten clever. And one of the things they do is that they hide their card-testing in the volumes of very large businesses. So, if you think about a very large e-commerce company, you can think about how many transactions are there. A card-tester might sprinkle, like, 100, 200, 3, 4, 5-cent transactions in testing. Traditional ML is, like, not going to catch that. Then you have a foundation model. Each, you know, charge becomes this, like, dense embedding. You start to see these clusters sort of pop out and you know in real time that they’re card-testing and you can block them. So, yes, it is happening on the charge path in less than 100 milliseconds of latency.

Alessio [00:06:15]: Have the foundation model enabled more data to be put in the embedding? I think, like, you know, things like, you know, number patterns and, like, zip code versus, like, location. I think those you could do before. Are there any new data points that you get? Yeah.

Emily [00:06:28]: So, I think there’s... I think there’s two big things. Like, one, you know, when you’re building a small model, you’re usually, like, oh, looking at the data that, like, has reasonable labels. It’s, like, recent history. You probably have some, like, hand-engineered features in many cases. If you’re actually, like, building an FM and you’re imagining there’s many downstream use cases, you’re putting, you know, tens of billions of transactions in it. You’re putting, like, the entirety, like, every detail of the payment in it and letting the FM run. And you’re putting some reason about what are the components that matter versus not. So, it is literally all the things. But I think what’s even more interesting is, like, the last K. Like, what matters is the sequence. You can think of a payment sort of like a word. And so, you can think of payments data kind of like language data. And what matters isn’t the word, right? It is the word in relation to the words around it. But what’s tricky about payments is you don’t see, like, you know, Emily on a podcast saying 20 words. And no, those are not 20 words. Like, the sequence that could matter could be, you know, this particular retailer charges from this IP. It could be anyone on a Friday night with this credit card. And so you kind of have to choose a broad swath of relevant sequences to capture kind of the last K that matters. So if you think about like a movie, right, like what are the scenes in a movie that you need to be watching to know if there’s something anomalous happening?

Swyx [00:07:57]: Yeah. And for listeners, I think you’ve talked about this in a number of places. The card testing detection numbers went from 59 to 97 percent.

Emily [00:08:05]: Yeah. On large users. Which sounds pretty helpful. It was really helpful. But the other thing that was really helpful was like the speed at which we got it out. Right. So like we had a couple of the large AI companies came to us and they said, hey, this is like after card testing, they’re like, hey, radar is amazing for finding fraudulent disputes. And that’s what it’s that’s. That’s what it’s trained on. Right. Transactions that result in fraudulent disputes. But we have all these sus, like suspicious transactions that don’t result in fraudulent disputes, but we still want them flagged. We want them flagged because even though they don’t result in a fraudulent dispute, even though we get paid for them, like they’re bots, it’s not good traffic, like they’re messing up our numbers, all sorts of different reasons. We can talk about some of the fraud that AI companies are facing. And so we want you to send us a pipeline with all the sus transactions, even if they’re going to be revenue generative because we’re probably going to want to block them. And it was like literally days like, OK, like FM embeddings, clusters, you know, good textual alignments. You can start to label them and you’re like, this is the clusters, you know, that look sketchy because they are enumerating some component of the login flow. These are the these are the ones that look sketchy because they’re enumerating some components of the login flow. These are the ones that look sketchy for X, Y, Z, other reasons. And then the AI companies could literally say, OK, this batch we want to block this batch. We don’t. So it just allows you to move faster on identifying not just new fraud vectors, but like whole new types of suspicious transactions.

Alessio [00:09:35]: How has the scale changed with AI? So before, you know, I used to run some software website and we would have the same issue. People buy the software and then gets charged back. But it’s like 20 bucks. Like today, you could like, you know, use the credit card and sign up for the OpenAI API and spend $10,000. $15,000. Like what’s the shape of the fraud today?

Emily [00:09:55]: So friendly fraud is like not stolen card credentials, but something like nonpayment abuse, free trial abuse, refund abuse. So it’s me. There are my credentials, but I’m not actually creating a creative revenue for the business. This has happened for a while. And actually, if you if you ask business leaders like I think something like 47 percent payments, there’s like 47 percent of them will say that their biggest. Fraud challenge is friendly fraud. I would say this was like just much less of an issue for SAS for two reasons. One, like what were you stealing? You weren’t stealing computer inference or whatever. And two, more importantly, even if you were stealing some service, like the marginal cost of providing that good or service for Salesforce or whomever was like near zero. And so it didn’t totally crush your unit economics. Now we’re in the world where GPUs are expensive. Inference costs are high. And free trial abuse. Free trial abuse or refund abuse or general nonpayment abuse. Right. You you rack up these charges and you never pay is like existentially threatening for AI businesses. I was talking to a small AI founder the other day because we’re building sort of a suite of radar extensions that are explicitly targeted at this type of fraud. And everyone tells me it’s a huge issue. And so with every company I talk to, I try to dig in on for you. What exactly is the issue? And there’s the first guy who told me it’s not an issue. And I was like, oh, fascinating. What are you doing? He’s like, well, I. I completely shut down free trials and I dramatically throttle credits until you’ve proven ability to pay. And I was like, well, you don’t think fraud’s an issue, but it’s like totally like you’re choking your own revenue. Right. So anyway, we worked with it. We got free trials back on. And that’s that’s in flight. What’s interesting. This is definitely a problem for AI companies because of the marginal cost. But it’s not only a problem for AI companies. If you think about like advertisers. Right. If you’re like you’re like a social media platform. Right. Advertisers come in, you let them start advertising. They do post hoc billing. Right. So you rack up some spend and then you pay. And if you don’t pay, that actually is expensive. Not because you’ve sold on compute in this case, but because you’ve taken ad slots from businesses who would have paid. Fascinatingly, I don’t know if I should mention. I think I could say it. OK. So my friend got a Robin Hood credit card the other day. Yeah. And literally as part of getting the Robin Hood credit card, he was marketed that he could also get. Free trial cards and I was like, oh, tell me more. What are free trial cards? Free trial cards are basically like cards with your name on them that are good for 24 hours and then expire so that you can sign up for free trials without ever having to get charged. So in the hands of a well-meaning consumer, that sounds fine. In the hands of a fraudster, that’s like extremely disruptive to the AI economy. We just announced our free trial offering. We can catch the majority. We can catch the majority of free trial abuse at the source. We’re working on the analog for refund abuse. One of the places where refund abuse is really painful, you mentioned this in the context of large volumes, like some of these AI companies will have like enterprise grade plans where it’s like $600 or $1,000 or $10,000 a month for hundreds of thousands of credits in whatever units they’re providing. And those are the ones that are getting hit with refund abuse. So a lot of the free trial abuse is like the consumers, the little dollars, but it adds up. But a lot of the refund abuse. is like very, very large subscriptions. You use it in full and then you go and cancel. And we see the usage happening. So like, and we can verify that it’s the person. So there’s a lot we can do here. And I think we can burn it down, but it’s like clearly creating a lot of pain for the ecosystem today. And just that vignette of that founder who told me he’d solved fraud. I just, you know, I think that just speaks to like, it’s so painful. They will literally give up revenue to not have to deal with it.

Swyx [00:13:49]: I think you teased a little bit. About how you’re extending Radar to serve new AI business models. And I think Stripe in general, I think is interested in like enabling payments for, for these AI business models. But basically like what, what do people want and maybe what’s realistic versus what is not realistic. I don’t know if that’s a term.

Emily [00:14:10]: For sure. So, you know, I think of Stripe as like the skeletal system for AI companies. So if you look at the Forbes AI 50. All of the. All of the Forbes AI 50 who monetize online, monetize through Stripe and what do they use us for? You know, most of these companies, I mean, you’ve seen it with like the, the cursor and lovable examples, right? They build these very scaled businesses with very lean teams. And so they want to go like all in kind of on Stripe to get many layers of the sort of economic infrastructure, financial infrastructure stack in one go without needing to hire humans to do it. So they use us for payments. These. They use us for payments. We were looking at the top hundred grossing AI companies on Stripe. And like the median was in 55 countries at the end of their first year and over a hundred countries at the end of their second year, which is like twice as global as the SAS wave from three years before. So they, they almost all adopt our optimized checkout suite, which comes with a hundred payment methods out of the box, very global reach. They almost all adopt radar and our fraud suite. And then one of the things that’s been really interesting is, you know, I think the, the market’s still trying to figure out what’s the intersection between supply and demand. And so there’s a lot of iteration across monetization models. Like, is it a fixed fee subscription? Is it pay as you go usage? Is it this credit burn down model? And there are revenue implications. There are also fraud implications, but equally important. There’s like unit economic implications. And I think one of our recent ahas was, you know, as the LLMs have gotten better, more and more AI companies are rappers and I don’t, I don’t say that in a, sorry, rappers are not an R and I don’t say it in a derogatory way. I say it in the same way, Arvind Srinivas once said to me, like, I am proud to have started as a rapper, product market fit and build an amazing product and move really quickly and not get slowed down in the research and other people could provide the underlying models. I could do that later if I wanted or not. But yeah. But because a lot of these AI businesses are rappers, right? Their services have an inherent LLM cost underlying them. We know that LLM model providers are ebbing and flowing. The underlying models are getting better or worse. The price of those models are getting better or worse over time. And so that actually leads to a lot of complexity in how you price your final service. When that final service is like so dependent on an upstream LLM. So one of the things we launched two weeks ago now, we call it token billing, but it’s basically you, it’s an API that lets you track and price to inference costs in real time. And what does that do? It’s like, well, if, if your service is built on underlying LLM and the model cost drops 80%, which we’ve all seen it happen, right? You don’t want to keep your price where it is because the competition is going to swoop in. But then conversely, and I think more threatening if the, if the cost of the underlying LLM three, Xs, which it also sometimes does, surprisingly, you could have unit economics that are literally underwater if you don’t, if you don’t adjust your price. So, um, you know, token billing is an example, but, but we’re seeing these AI companies, you know, iterate across usage-based billing, outcome-based billing is kind of an interesting one.

Swyx [00:17:29]: Does Stripe get involved there because that’s not really within your normal wheelhouse.

Emily [00:17:34]: We do. So our, we have, so we have payments and then we have a billing suite and almost all the AI companies use our billing suite. Billing includes. It includes things like fixed fee subscriptions, but it also supports usage-based billing. So like metered billing, um, and you can define usage in all sorts of different units. And we have a number of customers, including Intercom who actually define it in terms of the outcome. So in this case, it’s like support cases, resolutions, exactly. And, you know, I, I think it’s really interesting. I’m an economist by training. I told you earlier in the elevator that I think physicists make the best, uh, scientists or MLEs, but I didn’t know that at the time. So I’m an economist. Um. And I think a lot about, okay, like what makes the market efficient or inefficient. Um, and one of the things that I worry about in AI is it’s incredibly hard to take a product to market when someone has to pay for it before they see the value. Um, and that’s especially true with AI because a lot of the buyers, especially enterprise buyers don’t understand how to evaluate the underlying technology. And so if you can get your foot in the door by saying not just, oh, you’ll only pay for what you use. I mean, pay for what you use. I mean, pay for what you use is kind of helpful because they’re not committing upfront to some huge contract, but they can come in with a fear like, well, what if my employees use it a lot and it’s not actually helping the business. And if you can come in with like an actual cost sort of pricing function that is clearly profit positive for them, um, it’s a lot easier to get your foot in the door, right? So, you know, that a human resolving a support ticket is X, I promise that I will charge you, you know, less than X, like it seems conditional on quality strictly better. Uh, for you to try out my service than not. So anyway, we see, we see a lot of outcome based billing. The other thing we see an interesting amount of from AI companies is stable coins. And this one’s, this one’s earlier, but their use cases are interesting. So like if you, if you go to V zero now and sign up for an account, you can actually pay, uh, in stable coins. We’re seeing this a lot for AI companies that want to have very global reach, but also for AI companies that have very high price points. So like shade form, the, the YC star. Up is a great example. Um, they accept stable coins. Stable coins are now actually like 20% of their volume and their use case for stable coins is basically like very global and very high cost. And so the global means ACH isn’t an option, right? ACH is usually what folks in the U S would go to for low costs. If you’re going to use something like, you know, international cards though, on like a very large basket costs, you’re talking about paying four and a half percentage points, literally. Just to international card costs. And so that’s just taking a bunch of your margin that you don’t, that you don’t want to give away. So now 20% of the volume comes through stable coins. We actually did an experiment with them and half of that is fully incremental, which is to say they would only have 90% of the revenue they had, had they not opened up stable coins. The other half is a shift from other payment methods to stables. And then on the cost side of the house, you know, the cost of stables for them is like 1.5 percentage points versus 4.5 percentage points. So that’s a couple extra percentage points. So stable coins is another thing that we’re seeing AI companies adopt pretty quickly. I like to say that nerds buy from nerds and so there’s a nice sort of network effect there, right? Like if people who use AI have stable coin wallets, when they go to the next day, I provide her, they have a stable coin wallet, right? It’s sort of self-reinforcing. We also see this with link, which is our consumer product. Like link just passed 200 million consumers. So it’s not a small network. But what I think is more interesting is in the case of AI, it’s a very, very dense network. So lovable accepts link 58% of lovable’s volume flows through link. So for every three people who are buying on lovable, two of them are buying with one click link checkout because they already have a link account. And I think that just like gives you a flavor of, of the density of the link network, but also the density of the AI network. Yeah.

Swyx [00:21:37]: Is there a, are there classical measurements of network? Is there a network density that you keep an eye on, even though it’s obviously as an economist, like that’s the first thing I go to. Yeah.

Emily [00:21:46]: Um, I mean, the Herfindale index is less about network density in particular, and more about, as we look at all of the transactions that are flowing through Stripe, how concentrated are they on? I mean, you can look at it along a lot of dimensions. You can, and merchants, how concentrated are they on certain merchants versus spread across merchants? Yeah. How concentrated are they on certain industries versus spread across industries? How concentrated are they? How concentrated are they on certain geos versus a broad range of geos? And yeah, we definitely track that concentration. Now for us, some of that is actually the inverse of network density, which is we want diversification, like, you know, um, and we want to be exposed to many different industries and many different markets and have global reach because the current wave is AI and I am incredibly bullish on AI, but we really want to be growing the GDP of the internet broadly. And that’s not constrained to only the AI domain. Yeah.

Swyx [00:22:38]: Um, excellent. Should we, uh, move into the ACP? Uh, I don’t know how to transition it better than that. I agents do want to eventually do commerce between themselves. I guess that’s the transition and that, that, that, uh, is that the perfect intersection of financial infrastructure and AI. So maybe, uh, could you tell us the story of ACP, right? Like, I think, uh, this is one of the, the, the biggest launches of, I guess, like in the, in, in, in the second half of the year and, and like, I guess a really important strategic move between OpenAI and Stripe. Yeah.

Emily [00:23:09]: So, you know, we talked a bunch about AI companies in general. One important slice of AI companies is AI commerce, agentic commerce. And, you know, I think just zooming back, like we’re all spending more and more time in some combination of broad consumer-based tools like ChatGPT and AI dev tools like Replit or Vercel or whatever. Right. Right. Right. Right.

Emily [00:23:43]: And I think, you know, we saw an early version of this in ChatGPT with Operator, but an important area we want them to take action is buying on our behalf. You know, sometimes it’s recommending products, but often it’s like literally getting it all the way over the wall. So, um, a couple of weeks ago, we announced our agentic commerce protocol, which is joint with OpenAI, and it’s basically just a shared standard for how businesses can talk. So if you think about it, like it used to be that a human was buying from a business, now there’s an agent that’s sitting in the middle and that fundamentally needs to change how the financial infrastructure works, like checkout needs to look different, fraud checks need to look different, payment flows need to look different, but also merchants are trying to figure out how they can efficiently expose their product catalog, their inventory, their brand, their pricing through a range of agents to have access to their products. So it’s kind of a brave new world. So the agentic commerce protocol is really about that shared language for agents to get from merchants what products they have available at what prices, how they want the brand to appear. And then we also built a shared payment token, which basically allows the agent to pass over the required payment credentials on behalf of the buyer, because the agent doesn’t want to bear the risk. The agent doesn’t actually want to be in the middle of the transaction. And the merchant wants to, you know, undertake the charge and actually have the direct relationship with the consumer at the end of the day for returns and more. And so the shared payment token was also an important component. And then fraud was another important component, right? Is this a good bot or a bad bot? There was a day not very long ago when the optimal thing to do as a business was to block all bots. Now, many bots are good bots. You do not want to cut off that demand. And so what we pass over as part of the shared payment token includes scores on the goodness of the transaction so that the merchant can make the right decision. One of the ways this manifested was an instant checkout in ChatGPT. Have you guys bought anything from this?

Swyx [00:25:55]: Not yet, honestly. I’ve tried. It just recommends things, but I always want to take over the last mile of checking it out myself. It’s hard for me to, like, hand over control.

Emily [00:26:05]: Okay, yeah. And I think they’re also still, like, iterating on their recommendations. To some extent, last night, my daughter was at a class, and so I took my son out to dinner, and he told me he has a school play, and he is supposed to dress as a Spanish shopkeeper. And so I tried to search for, like, kids’ Spanish shopkeeper outfit, and they recommended me a $1,300, like $1,300 bolero off Etsy.

Alessio [00:26:29]: The play is not that important.

Emily [00:26:31]: I love the child, but the play is not that important. But there is a lot of great stuff you can buy. And so in the... In the initial Instant Checkout Lodge with ChatGPT, you could buy from U.S.-based Etsy sellers. There’s over 1 million Shopify merchants coming soon, including some really big ones like Glossier and Viore. This week, Salesforce announced that they’re also in. And then, you know, my favorite is that, like, a week ago, two weeks ago, you could have asked, hey, are the largest retailers going to get on board with this or not? In the last couple of days, Walmart and Sam’s Club have just signed up to also make their inventory purchasable through ChatGPT and the Agentic Commerce Protocol, which, like, I don’t think that there is a bigger signal on a big retailer being up for it than Walmart. The world’s biggest one, yeah. Yeah, yeah, yeah. So that’s pretty exciting. And then, you know, one of the things that is important to us and to the broader ecosystem about the Agentic Commerce Protocol is, is it’s not about Stripe. So that shared payment token I talked about or the Agentic Commerce Protocol, like, that works no matter who your payments provider is. We can pass the shared payments token over to any other PSP. You don’t have to process on Stripe. It’s also not just about OpenAI. And so in the same way that, like, you and I are seeing, you know, new models come online all the time, we want to be able to move across models flexibly, there’s going to be sort of new agentic buying experiences coming online. And we want to make it easy for merchants and kind of, like, one shot to integrate with all of the agents as they come online. And that’s what the ACP really provides, because it’s a standard protocol versus needing to do custom integrations per agent. Did you guys see Karpathy’s tweet about the, he basically, like, recreated…

Swyx [00:28:27]: NanoChat? Mm-hmm. Okay. Yeah.

Emily [00:28:28]: So, like, if he can do that, 8,000 lines of code, less than $100, like, you might think that a lot of companies are going to roll their own really soon.

Swyx [00:28:36]: Using NanoChat? That would be interesting. I haven’t made that connection yet. Let’s see. Let’s see.

Emily [00:28:42]: But I think this basic premise that, like, it may be a winner-take-all, but it’s not yet clear who the winner is. And by the way, like, I hope for the efficiency of commerce that it isn’t a winner-take-all. Therefore, many merchants need to actually be having their products sold through many agents is kind of the premise of ACP. And we’re just delighted to see the early traction. And, you know, have just been flooded, honestly, with both merchants and AI platforms wanting to join. So, I think we’re on the right path.

Swyx [00:29:15]: Yeah. And this is your protocols, kind of. Yeah.

Alessio [00:29:18]: We did an episode with Crunch AI, which does web rewriting for agents. And they have him as a customer. They have scams. And I think every brand is, like, I mean, for brands, it’s very easy. It’s like, hey, I don’t really care where it comes from. Yes. If you buy my thing, we’re friends, you know. And I think, like, you take away. You take away the part which is, like, the most annoying to think about, you know, which is, like, the fraud. I do have a question on, like, the good bots, bad bots when it comes to, like, scarce releases, like tickets for events and things like that. I think that’s going to be interesting. Oh, please, Salt Ticket Master. Oh, my God. Exactly. It’s like, well, but now if anybody can just have an agent that just goes on the website to buy them, now it’s like, how do you do the queue? Because everybody gets there instantly, right?

Emily [00:29:59]: I won’t say the company, but I had dinner the other night with the guy who’s the CEO of. You can basically. You can basically think of it as, like, StubHub for Country X. I won’t say what Country X is. And he was selling, I think it was, like, 3,000 Bad Bunny tickets. And he had 400,000. Oh, my God. People come to buy the Bad Bunny tickets, except almost all of those people were actually bots. And this is a great example of what we were talking about earlier, around, like, it’s not just about the fraudulent dispute. So the conversation I was having, he was like, you know, we have scalpers. Scalpers who have bots. They end up scalping the tickets later. They come and they buy. It doesn’t result in a chargeback. Like, they pay, but they’re not the people that we want paying. And so to our conversation earlier on suspicious transactions, like, there are lots of different types of fraud. And thinking of fraud as just things that result in a fraudulent dispute is actually overly narrow. And he wants to block, you know, all the scalpers, everyone who’s, like, you know, enumerating through email addresses in their signup and, and, and. The other thing he said to me, which was interesting, is he. He, for various reasons, and some of this is actually, like, the, the, the nuances of how his system is built. But I think some of it generalizes. He wants to have those fraud signals before they even get to the checkout page. And so how can we understand the customer independent of them entering their payments credentials? And there are a bunch of ways we can, and we can get better there. His particular reason why is he’s got, this is a little mundane. But once the ticket goes into the cart, it can’t be touched for, like, 10 or 15 minutes. And so even though, you know, it’s successfully blocked at the actual charge time, it’s, like, been held from, from those, those other good customers. So anyway, I, I think there’s this, there’s a lot of exciting work to be done that’s actually, like, increasingly possible. Not just because of the scale of the Stripe network, but also because of AI around understanding an expansive set of fraud vectors. And, you know, if, if you think about traditional. Terministic systems, you know, you’d write rules like block, don’t block. Now you can think about, okay, actually foundation model, text alignment, like human readable description of like why we’re worried about this charge. And then today, a human, tomorrow, an agent sitting on top of that and decisioning, like reasoning over, the model outputs. Um, I think that’s the world we’ll be in, in the next three to six months. Yeah.

Swyx [00:32:28]: I think we need to. We have to be careful about rolling those kinds of things. Yeah. Because people get very upset and justifiably so, and they’re denied something that they, they, they should have by a bot that won’t explain itself. Yes. Right. There needs to be like an appeals process or like something like tier two human, like.

Emily [00:32:45]: For sure.

Swyx [00:32:46]: Can I speak to the manager, please? For sure.

Emily [00:32:48]: And yes, for sure. And well, humans make bad calls too. Sometimes they make bad calls at higher rates than LLMs because they can’t reason over as much information, but I agree. Definitely appeals process. But then also like. When we actually look at. Bad actors, it’s like a tiny, tiny share of bad actors accounts for a huge volume. Right. You’re just rooting those out. Of the bad outcomes. Yeah. And so what ends up happening is actually good actors have worse experiences, which could mean they don’t have access to free trials or they’re gated and how many credits they can use prepayments or they’re just charged a higher price because they’re covering for the cost of the bad guys. Right. And I, and I. And I actually think there’s an opportunity to like, you know, I’m from Montana, so sometimes I talk about sheep and goats, although I don’t actually know which is good and which is bad. Separate the sheep from the goats in a way that’s like really good for the good actors.

Swyx [00:33:44]: Why are sheep and goats bad? Like.

Emily [00:33:46]: No, only one’s bad and one’s good. But I don’t know which is. Why can’t they both be good? That was like an idiomatic expression everyone used growing up to separate the sheep from the goats. And I’m like, I don’t know. But they’re both good. They’re both cute. I think sheep are supposed to be cute, but I’ve always been impressed. Have you guys been to Yellowstone?

Swyx [00:33:58]: Yeah.

Emily [00:33:59]: You go into Yellowstone National Park, like the mammoth entrance, and there’s this like sheer giant, you know, wall of rock. And there’s these mountain goats. The goats just chilling. Exactly. They’re like scaling it at 90 degrees. Yeah.

Swyx [00:34:10]: Like mountains I get, but like dams, I don’t know why they love dams so much. It’s like, it looks so dangerous. And like there’s babies just walking on there.

Emily [00:34:17]: So dangerous. Yeah. It almost looks like AI generated the image, but it’s real.

Swyx [00:34:21]: It’s real. It’s real. And okay. I’m going to do like economist corner just because like, you know, you clearly still identify as an economist.

Emily [00:34:29]: I do identify as an economist. It’s very strange. Yeah.

Swyx [00:34:32]: Yeah. Yeah. Well, so like when you encounter those, like the StubHub for X, don’t you feel the temptation to recommend an auction? Like, and there’s so many auction mechanisms that can clear the market. This is clearly a market clearing problem. What’s the market solution here?

Emily [00:34:46]: I don’t know that I am usually tempted to recommend an auction. I am usually tempted to ask a lot of very probing questions about why they’ve designed the system the way they’ve designed it. Yeah. Yeah. Whether there’s a, whether there’s a more efficient path, but yeah, I, I feel that way about, um, most, most pricing matching discovery recommendations. Like most markets are just inefficient. And so I think there’s a lot of opportunity to make it better. Um, one of the reasons I joined Stripe and one of the things I have loved about being at Stripe for the last four years is when we see those opportunities to make the market more efficient. Um, we can, we can actually invest in doing them without optimizing them, you know, without monetizing them directly. So you can think of it like incentives are very aligned. Anything we do to help the businesses on Stripe grow helps us grow because, you know, they, they run their payments through us. Yeah. Um, and so we do this all the time. Like, here’s how to, you know, improve your checkout. And we just like update the checkout for them or like optimizing their payments acceptance or automating their retries. Um, so anyway, I think it’s just, it’s, it’s very nice to be at a company where you don’t have to worry about the go to market for something that helps the businesses that run on you. You just have to help the businesses that run on you do better. Um, and that in and of itself is, is a good outcome for your company and justifies the investment. Yeah.

Swyx [00:36:15]: They’re very incentivized. It’s like top line and bottom line sometimes. Yeah.

Emily [00:36:18]: And by the way, like this isn’t all causal, but, um, last year the companies on Stripe grew seven times faster than the SMEs.

Emily [00:36:31]: And, you know, again, like not all causal and there’s selection effects, but definitely some of it, some good, some good tailwinds. Yeah.

Swyx [00:36:38]: Yeah. So, I mean, yeah. And I think the economist term I really like is deadweight lost and like basically like eliminating friction. It means improving surplus for both producer and consumer and Stripe benefits as a result. It’s just like, that’s nice. Everyone wins. Uh, it’s a, it’s a good, good, uh, fuzzy feeling. Coming back to the protocol. I think that it’s an interesting decision to actually release it as a protocol. Like you said, it’s going to be many to many. And like sometimes Stripe is not involved. You also mentioned like Stripe link and Stripe checkout, and those are Stripe products, right? Those are, those are not protocols. So I think like it’s a very interesting and pivotal decision to choose to release it as a protocol as, as opposed to not. I was wondering if there’s like any internal debate or is any internal color about like the decision behind choosing a protocol. Yeah.

Emily [00:37:21]: Um, you know, Stripe has moved fast for the entire four years. I think it is accelerating and I think it is accelerating because customers, users changes in what the market needs, both what businesses need and what buyers need, um, in the world of AI is accelerating. And so new products, new solutions like ACP, like token billing are literally being pulled out of us. And what we saw was a hole in the market, like consumers, and we can talk about developers too, but like consumers want to be buying through ACP. Agents, agents are ready to buy for them. Merchants are ready to let agents buy on behalf of consumers. And yet the market can’t figure out how to make it work. And that’s not about Stripe. That’s about growing the GDP of the internet. That’s about making sure commerce can flow. And, you know, so, so there was no debate. It’s like the ecosystem needs this. Like we will pair with open AI to get it over the wall. And, and by the way, like, as I said, we’re early days, you know, ACP is seeing a ton of traction, but. If it wasn’t, or if something different comes out six months from now, like all we want is a shared standard. We don’t need to have like our name on the shared standard. I think it’s absolutely, absolutely the right thing, the right thing to be doing. You know, I have had folks ask me like, is agentic commerce, just a straight substitute for the commerce that’s already happening on the internet. Yeah.

Swyx [00:38:47]: It’s just like fancy APIs. Exactly.

Emily [00:38:49]: And I think the answer is it’s not a straight substitute. I think it is actually like expanding the aperture of what. What commerce will get done. And the first person to put this bug in my head was Dwarkesh. And when he said it, it actually like took me a second, like, I didn’t believe it, but I think it’s right, which is if you look at the share of income that is spent on consumption as a function of how much income you have, you see that high income people spend a much lower share of their income, low income people spend a much higher share of their income. There are many reasons for that. But one reason is the biggest cost to very high income people consuming is not the dollar cost. It’s the time costs of consumption. And so I’m very interested in how agentic commerce can open the aperture for spending by high income people because it’s removing the most costly or binding constraint, which was their time. And we’re actually then truly pumping incremental, not substituted, but additive dollars into the economy. And obviously, like, you know, for. Yeah, well, it results in sometimes buying $1,300 costumes and I would rather spend an hour than buy a $1,300 costume.

Swyx [00:40:08]: Well, you know, just work a few more years to stripe. You’ll get it. So so I think one like there’s some I think interesting where I love protocols, you know, as a developer tools person, I’ve been I’ve been involved in designing a few of them, debating a few of them. What are the forks in the road that, you know, like someone else was like discussing? Something really strongly, and we decided against it, or maybe it’s still an open question. I’ll give you one and then maybe you can volunteer another. So the I mentioned that both Solana and Circle have sponsored my conferences before, and they’re also trying to be that build a protocol for agents and both of them actually give agents a wallet, right, as opposed to a payment token. And I think like having an agent with their own bank account effectively is an interesting choice. You didn’t go for that. So like, is that a decision factor or is or is there a different one that you want to focus on?

Emily [00:41:00]: Let me parse two things. When I think of the commerce protocol, that’s primarily around what’s the standardized way that businesses expose their products and their inventory and their prices and their brands and make those available to agents to expose to the consumers and or to buy on behalf of the consumers. That really comes down to like, how should a product catalog be expressed? How should prices be expressed? I think the current version is like the bare bones version, and it will continue to evolve. For example, like you could imagine like from a market clearing perspective that the merchant should also be as part of that articulating the cut that they’re willing to give to the agent, right? Like, like a little bit. Yeah.

Swyx [00:41:48]: In a classic, like if the human agent, I give them a budget, like what their negotiation target is and what their max spend could be. Yeah.

Emily [00:41:55]: Yeah, exactly. And so in this case, I’d be like, okay, well, you know, whatever, the Bolero that I didn’t buy is a terrible recommendation, but they have many good recommendations, but that was a terrible one. You know, this costs $1,300 on Etsy, but, you know, I’m willing to give X to any agent who facilitates the transaction, either on top of or underneath the $1,300. So anyway, I think there’s going to be an evolution of like the various parameters that should be included. But like the, the basic set was, what do you have to be able to deterministically expose to an agent so that they understand what’s available and what representation of the product and brand and, you know, sometimes it’s size and number and whatever, like has to be made available to the agent and or the, the, the human that’s initiating in the first place. Okay. Okay. So shared payment token is a little bit different, which is like, okay, how do you actually get the transaction done? Like, how does the money flow? And even at Stripe, like that has been evolving. So shared payment token is, is what we built and launched and have in the background of the instant checkout implementation with ChatGPT. But a year ago, Perplexity launched a travel search and booking agent. Did you guys see this? Yep. That is also powered by Stripe. And there, the payment flows are a little bit different. So we have an issuing product, which allows you to issue virtual cards. And what happens in that sort of flow is the agent gets issued a one-time use virtual card to spend on your behalf. And, you know, people get very, get very jumpy about that. But I like to remind them that like, when I order from DoorDash, my Phil’s coffee, right? DoorDash is issuing a one-time use virtual card to the, for $6, believe it or not, to spend on my behalf there. And so now you’re just inserting, you know, AI agent instead of human agent. And in the same way that my DoorDash driver never saw my card credentials and couldn’t spend, you know, more than $6 and had to spend it in a constrained time window near my house, same thing for the AI agent in the Perplexity travel search and booking agent. And we still have agents doing commerce through Stripe. Using the virtual card implementation and their pros and cons. So I don’t think that like, it’s going to be all virtual cards, going to be all shared payment tokens. It’s going to be all agent wallets. I think stable coins will be an interesting direction. I think wallets in general will be an interesting direction. I think stored, stored balances will be interesting, especially as we’re talking about kind of micro transactions, right? So we’re talking a lot about buying like goods, buying goods are usually priced high enough that it’s worth sort of like a car. Card transaction type approach. But if you’re talking about buying AI or buying some inference or buying content, you want to be able to make 5, 10, 25, 50 cent transactions. And those are hyper inefficient in the card world. So I think agent to agent payments are going to push us a little bit to the next frontier here. But ACP, again, is like distinct from how the shared payment token works or how the money flows. And I think that will, A, continue to evolve just because the market needs are evolving and the technology possibilities. Are evolving, but B will also like also doesn’t need to be standardized in the same way.

Swyx [00:45:28]: What about the receive side? I guess, can my agent make money for me?

Emily [00:45:33]: Can your agent make money for you? Oh, I was thinking the opposite, which is like, I’d be happy to give an agent, you know, $3 to go out and do deep research for me. And so we’re trying to figure out how to enable that.

Swyx [00:45:42]: No, that’s research. No, I was like, you know, like spending, I think like there, we have a good mental model of how to spend, especially because we have human agents as well. Yeah. Because. How to spend, uh, making money is, is, is obviously the, the original draw of, of, of, uh, strive for any, any founder. Yeah. Uh, yeah. I’m just kind of curious. What’s, what’s your thinking there?

Emily [00:46:02]: We make it easy to monetize your MCP server. So that’s like one thing. We also are seeing an increasing number of new businesses get going in AI dev tools like Replit and Vercel. And so we want to make it really easy to spin up monetization also within those tools and within that flow, not be taken out of flow and go and create a Stripe account. Authenticate and whatever, um, a couple of weeks ago, we released claimable sandboxes. Have you guys seen it? We’ve been in Vercel and seen it or no.

Swyx [00:46:30]: I know that you have a sandbox product, but I don’t know about.

Emily [00:46:32]: So we have a sandbox product, which you’ve seen from like being within the developer experience on Stripe. Now that sandbox product can be like invoked, used, manipulated. You can create your products and your pricing and run test charges and generate customers. And in that sandbox. Before you even. Have a Stripe account, or maybe you have a Stripe account from your last business, but before you’ve linked it to this business, right. And, and we call them claimable sandboxes and Vercel and Replit were actually remember talking to Replit at Stripe sessions in May about our sandbox product. And they were explaining to me how, um, you know, people are trying to build businesses end to end. And like one of the wonky parts of the flow is setting up the payments integration and get going. And so we started talking. About like, okay, could you actually have like the sandbox environment there? Um, and it was whatever, four months later and they had it, they launched it. We launched it with them. And also, um, with Vercel, actually Guillermo had a cool post a couple of days ago that, um, our Stripe integration is now one of their top. I think it’s like the third highest, uh, integration that they’re seeing in V zero. Um, and we’re only two weeks into the launch, but it’s basically like all of these people aren’t going to create. V zero just for fun to create something, just a website or whatever, just for fun, they’re going to create a business. And so making it really easy for them to do the business payments back end part of that and how it actually works is literally like you’re right there in V zero and you, you know, you have your plant shop and you create your products and you set your prices and you run your test charges and you click a button at the end. Like when you like what you see, you can go back and iterate later, but you click a button and your tab opens in Stripe. And you can either sign into the account you have or create a new account and you claim that sandbox and that sandbox, you can take it live and it becomes your business. And lots of people are taking it live every single day. And we’re seeing new businesses get created, um, that were never before. And one of the things that’s really fun to see, I was going through the list of businesses. I probably can’t name them live, but like a chunk of them are AI companies, like a chunk of the startups being created today are AI companies, but a chunk of them are like non-technical founders who may have actually struggled. Like. Get going on Stripe had they not had it kind of within that V zero. Yeah.

Swyx [00:48:55]: Low code is a huge enabler.

Emily [00:48:57]: Low code is a huge enabler. And, you know, we’ve, we’ve done some good work in our own onboarding experience to make it low code and we have, you know, low code subscription and whatever. Um, we have our, our new onboarding experience internally, we call it future onboarding experience and it kind of walks you through what’s the business model you’re trying to create and then sort of, um, stands up the sandboxes for you. But what’s cool is now you do a bunch of that in say V zero, if you’re in V zero, and then you come over. And, and your future onboarding experience has already learned like your intent and your preferences and all of that from Vercel. And so you’re dropped in X percent of the way through, uh, with the sandbox already spun up.

Alessio [00:49:32]: So this is like from the outside, how people perceive AI as Stripe. What about inside? So you mentioned 3.5 was kind of like the moment you took it seriously. What were the first internal use cases and then how do you use AI Stripe today? Yeah.

Emily [00:49:45]: First internal use cases were, you know, bottoms up experimentation, right? So we created, we call it go LLM, but it’s like just a chat GPT, like interface where you can engage with a bunch of different models. Um, it was the very, very first version actually wasn’t like an LLM proxy where you could build production grade systems. It was literally just like chat GPT like stuff. And then we had this like preset feature, which was like prompt saving and sharing. And so you could share your tempo. Oh, you know, this is how I figure out what customers to reach out to and generate reach outs or, um, you know, rewrite. My marketing content in Stripe tone or whatever. And you had sort of hundreds of presets that came on like overnight because everyone was into it. And then we generated, so then LLM proxy was like, okay, now production grade access for engineers to these LLMs. And a lot of the early use cases there were actually around merchant understanding. So I mentioned a little bit ago, but we have like thousands of merchants that come on to Stripe every day and we have to understand who are they? What are they selling? Is it supportable through the card networks? Like, are they credit worthy? Are they fraudulent? Um, and there’s a. Lot that LLMs can do there. So those were some of our earlier, um, earlier use cases. Fast forward to today. I mean, you know, I, I actually was looking at the dashboard earlier cause I was planning for, for 2026 and some of our LLM costs, 8,500 stripes a day use our LLM based tools. Okay. There’s like only 10,000 stripes. Like not everyone is in every day. Like it’s basically, it’s basically everyone. Um, and you know, I think people are getting like pretty creative in, in the. Um, I was talking last week to the LPM teams, the local payment methods. Um, you know, you and I think a lot about cards or whatever, but local payment methods matter because the businesses on Stripe are almost always selling internationally. And when you’re in other countries having local payment methods, you know, Jiro pay if you’re in, um, yeah, I’m from Southeast Asia. Yeah. It’s all over. What’s your favorite?

Swyx [00:51:45]: Uh, well, I know. I mean, uh, there’s like grad pay, I guess. I don’t know. Yeah, exactly.

Emily [00:51:49]: And like, if you don’t. If you don’t see a local payment method, like if you only see a card, you might not have cards, or if you only see cards, you might not feel like it’s like localized to you or meant for you. Whereas if you see like in market regional payment methods, you feel much more connected. You’re much more likely to convert statistically. And oftentimes the fees also, uh, make more sense, um, for, for the merchant. So we’ve invested a bunch in integrating local payment methods. Most businesses on Stripe use our optimized checkout suite. Our optimized checkout suite comes with over a hundred payment methods out of the box. Right. Right. Right. Right. Right. Right. Right. And so, you know, what is the, the challenge? The challenge is integrating with any new payment method. And it takes like two months for a couple of engineers. It’s not the end of the world, but Stripe’s a pretty lean company and we got a lot of stuff to do. And so, you know, for the marginal payment method, is it worth it? Yes or no? Well, when you step back, what are you doing? You’re really like looking at Stripe’s code base. You’re looking at like how the LPM works. And like the integration guide for the LPM, and you’re kind of like hooking the two up. And so the LPM team, it took them two weeks for the first one, but they just launched a new pan-European payment method in two weeks using an LLM to like build that integration. And I think they’ll probably, you know, have it down to a day or two within, within a month. And so that’s just a good example of like, it’s kind of should be just like a machine talking to a machine and there’s pretty good documentation on both sides of the house. And so the LLM can make it. It can make it pretty far. We also use a lot of AI coding assistance and yeah, I would say like 65, 70% of engineers use them on the day-to-day. I have a really hard time understanding impact. Like I don’t quite know what statistic to look at. I don’t actually.

Swyx [00:53:39]: It’s not lines of code. It’s number of PRs.

Emily [00:53:41]: I don’t think it’s lines of code because I have in the last week been sent three different documents. Yeah. That I know were like at least partially written by an LLM documents, not code. And in all cases, they went back to the individual and I said like, I actually just want to see the bullets that you put in chat GPT or whatever. Not the, not the eight page document because I have a really hard time reasoning about the eight page documents and it sounds good, but I’m not quite sure it’s like connected to reality. And I feel the same way about lines of code. Like, I don’t really want more lines of code, just like I don’t want more pages of docs. Yeah. Yeah. So we’re watching that. And then also the costs of a lot of these coding models is actually like pretty non-trivial. And so as we’re planning forward to next year, we’re reasoning a bunch about like, where can we get somewhat more efficiency there given like, obviously it’s valuable and we want people to be using AI coding tools for sure, but, and we want to make sure that, that we’re getting the right returns for the business. And some of that is managing costs. And some of that is getting a clearer read on impact. Yeah. Because we’re just starting to get that value. Yeah.

Alessio [00:54:46]: How do you feel the social contract is changing? Like you mentioned, just send me the bullet points, right? It’s like I could have sent you the bullet points before LLM’s but you are making me write this memo, right? I feel like in a lot of organizations, there’s like a performative and part of it is like, you know, wearing a suit to an important meeting. It’s like, in a way it’s like, hey, I’m doing it.

Emily [00:55:04]: I don’t wear suits. I don’t wear suits. I hear some people do. Yeah.

Alessio [00:55:06]: You know, I’m doing it to show you respect. And in the same way, I could have just sent you this bullet point. So we’re going to have this meeting in shorts. Do you feel like with AI now, it’s like, okay, if you’re going to do it with AI, just send me the bullet points and we’re kind of like breaking through in a way.

Emily [00:55:19]: Yeah. So that’s really interesting. Okay. So I hadn’t thought about this before, but here’s my working hypothesis. Tell me if it tracks. What I care about is that the expert in the area, like they’re an expert in the area. Otherwise they wouldn’t be sending me a doc, right? The expert in the area has thought deeply. And what is writing, like actual writing, typing, whatever, it doesn’t matter. But like writing not with an LLM forced you to do, it forces you to think deeply. It forces you to structure your reasoning. I don’t know about you guys, but when I read a doc, when I write a doc, I’ve like read the doc like 50 times and thought about like, does this logic track? Are there gotchas I’m not considering? Like, is that the train of thought? Like, how might somebody else, look at this? And it’s not like that I wrote the doc to be performative and the bullets would have been better. It’s that the careful, thoughtful, arduous, time-consuming construction of the doc forced me to appropriately reason from first principles. And I think LLMs do the opposite. Like, oh, you just throw in the bullets. You don’t have to reason from first principles. And it sounds good. And people like, but I think that’s extremely dangerous.

Alessio [00:56:28]: I think that is true. But I feel like you, you still are not generating documents with AI. So because you are the type of person that uses the writing as the thinking, you still go through the process versus the people that use the AI still wouldn’t put that much thought into writing the long document anyway. I think to me, that’s really the thing. Same way we were talking about this for code yesterday in another interview, which is like the slot machine effect of like cursor and like these tools. But at the end of the day, you got to merge the PR. So you got to come up with something that makes sense for the business. Like with these documents, it’s kind of like, oh, I don’t know. I don’t know. I don’t know. It doesn’t, like, you just need to come up with the right ACP design. I don’t care if it’s 10 bullet points or like 10 pages. To me, I think like things are changing now because also people read more summaries. So it’s like, well, if you summarize my thing, then why should I write a long thing? I should just write a short one. So, I mean, I don’t have an answer. Just like interesting to

Emily [00:57:19]: see how, you know, you’re like, just send me the bullet points. Yeah. The primary thing that I, well, there’s many things I care about, but like a very concrete non-negotiable is LLM was used in the generation of this content. Please cite the LLM. Because my least favorite thing is to be like two pages into what I think is a thoughtful doc

Swyx [00:57:40]: and find the annoying, like space, double dash, space. And then as a double dash guy, I feel like

Emily [00:57:47]: I was writing it before LLM’s ruined it. No, that’s not what tips me out. That’s the only thing that tips me out. But you know what I mean? Like, I do think we all need to be careful about LLM slop. And then I think there’s like a societal behavior. Behavioral thing here too, which is like, we can’t turn our brains off. I mean, I actually think that like, what do LLMs make all the more important in the world? The ability to think and reason deeply. To like question, to like tell the machine what to do more so than like to do and execute the task because the LLM can do and execute the task. And so if you’re looking at LLMs and you’re like, oh, that means I don’t have to think deeply because they’re just going to do it for me, which is very natural because they do produce. Yeah. And that’s, I think, is just an extremely enticing output. I just think it’s like, I think it’s like risky for society. I mean, I think we’ve seen how like people who grew up on social media have like a lot of issues with attention. I think people who, I don’t mean attention trying to get attention. I mean, attention, like staying focused on a task. I think in the same way, people who overgrow up in their work life on LLMs risk under-investing in depth. And I think that’s particularly dangerous in a world where with LLMs, Actually, it may not appear this way in the moment, but like depth is the most important thing.

Swyx [00:59:05]: I would push back a little bit in terms of, I think I’m maybe a bit more slob friendly than you guys. Are you? Just because like slob comes from humans and slob comes from AI. What just matters is like when you sign off, when I send you the document or when I send the PR, I am signing off on the whole thing. I can’t abdicate responsibility to the LLM. Maybe LLM had good output, maybe not, but I’m the final judge, I’m the editor, right?

Emily [00:59:28]: So I actually think we’re on the same page there. So I am like all for using it in the generative process. Actually, it was really cool. It’s a tool for thought. It’s a tool for thought. And it’s a tool for rapid experimentation and rapid iteration. And I love to look at like a demo or a prototype that like I don’t want to see a doc on it. I want to see like the quick thing that you spun in whatever tool you use. Actually, when we were working on claimable sandboxes, I’ll never forget Vercel. Vercel sent us basically like the V0 of like how they thought it should be implemented, just like the UI. Like this is what we think the experience should look like. And it was extremely clarifying, like much more clarifying than hours of meetings and pages of design docs. And so all day long on the generative, but like you need to like deeply put your stamp of approval before you push the PR, before you publish.

Swyx [01:00:19]: I did want to double click a little bit on both, you know, like two primary use cases on RAG and writing code. Just one. Just I guess on like in internal information, there is obviously Glean, which we talked about yesterday, and just all the other internal code search tools. You guys use Notion as well. Is RAG still relevant? Is that something that’s like in active development or what’s beyond that? What’s like the frontier?

Emily [01:00:43]: So we’ve actually been leaning in really hard on, we call it tool shed, but it’s like a, it’s like an internal like MCP server that basically has access to like all the Stripe tools. Yeah. And, you know, what I like about that. And it’s managed centrally, like, you know, there’s, you know, it’s managed centrally and it plugs into, we’ve since killed that GoLLM thing I talked about. And we did a new like a, you know, implementation, like open source, like Libra chat situation, which is great. But it like hooks up to all those same tool shed, like, you know, MCP servers. And it’s got, you know, I mean, everything you would think like Slack and Google Drive and Git, but also like access to Hubble. So it can see like our data catalog and all the data and it can query the data and, and, and. And I think that’s. I think that’s been really powerful. I don’t think RAG is dead. I do think there’s an important name of the game around, it’s not just like the information that’s available in all those tools. It’s also being able to interact with all those tools, right? It’s like the tool calling. And so I think they coexist and I think they, I think they coexist together. Also, while tool shed is owned centrally, anybody can, so you can, you know, the, the Salesforce team can add Salesforce and, and, and. Because we don’t want to be blocked on some central team in order to have those tools exposed to, to the LLMs and to the agents for Stripes. Yeah.

Swyx [01:02:04]: You want to decentralize a bit. And then code wise, you know, coming on the code side, obviously closer to home for me, I think it’s also like an economist problem of measuring productivity. It is.

Emily [01:02:15]: Okay. Well, now you’re just making me feel guilty for not having cracked it.

Swyx [01:02:18]: No, no. I mean, it’s, it’s unsolved globally. I’m kidding. It’s hard. It’s hard.

Emily [01:02:22]: Yeah.

Swyx [01:02:22]: And that’s the thing. Like you, you, you’re looking at the cost and you’re like, oh, it’s pretty high. I don’t know. Maybe we like move to open source models or something, but like, you don’t know the productivity gains you’re off. You’re, you’re getting an interim and you have pretty expensive engineers. Like it’s, it’s hard to tell.

Emily [01:02:36]: Engineers are expensive engineers also like all of us, right. Are hyper-motivated to do the best work of their lives. And so there’s an important component of like, if the people want it, like there’s inherent value in, in providing it, right? Like when people have the tools that they want, are learning the things that they want. Like they work harder, they’re more creative, they produce better output. So anyway, there’s all this like sort of soft fringy stuff that I think is added value above and beyond the actual, you know, production output. And then there’s also like the learning curve, right? So, I mean, we think about this a lot, like when you launch a new traditional ML model or now like AI solution, right? Like if you over-focus on the results in the immediate short-term window, you really risk getting a false negative. Like it’s not good enough. It’s not tuned yet. It doesn’t have the feedback loop yet, you know, hasn’t had time to get the training data to get better. And I think that’s like kind of particularly true in, you know, AI, because you, when you’re working with LLMs, it’s like, well, with like GPT-4, like it didn’t work. And then you’ve swapped in GPT-5 and all of a sudden it does, or we use like a GPT-4-0 and it was like kind of a little bit expensive to justify the humans that it was replacing for a particular risk-related task. But then next thing we know, like O3 Mini is out and it’s like, you know, $3 million a year savings for the business, both because the model is less expensive, but also more importantly, because it replaces more of the humans. And so I do think that like when I think about the optimal adoption of these AI tools, it seems wrong to focus on in-year ROI. And it seems right to focus on two-year, three-year ROI. Now, enhance. It’s inherently hard to know what two or three years is going to look like. But if we look at the history, models getting much better, much more quickly, models getting quite a bit cheaper, quite quickly. It overall makes me bullish that we shouldn’t over-obsess around in-year returns. Yeah.

Swyx [01:04:45]: In-year returns. That’s a good term I never thought about.

Emily [01:04:48]: Especially when the economy is doing so well.

Swyx [01:04:49]: Advertising it like that. Yeah.

Alessio [01:04:51]: What about data? What about data? I was going to move to the data side. Wait, what about data? Yeah. You know, it’s like, oh, are the engineers more productive or not? Like, what about, yeah, like, I mean, text is equal, right? It’s kind of like the first iteration of this. Like, what’s the productivity like on like, I mean, you can generate any chart now, right? But like, does it mean it’s good?

Emily [01:05:11]: Yeah, yeah. So we have this, he’s not really a guy. He’s an AI, but his name’s Hubert, which sounds like a guy. So we have this guy called Hubert, which basically is like natural language to ask questions about the business. By the way, we have a Sigma assistant. So like, our users can query stuff about their business on top of Stripe data. That’s a much more constrained problem because your Stripe data is like your revenue data. It’s like very well structured. It’s very well documented. It’s available in the dashboard and in Sigma and in Stripe data pipeline. You can hook it up to whatever. And so, you know, a text to SQL experience sitting on top of that, like, it’s not going to be perfect, but like, it’s pretty airtight. And by the way, like, if you use natural language to describe what you want, we write the query. And then we also tell you in natural language what the query is. So even if you’re non-technical, you can validate it. Okay. Now imagine there’s a lot of tables at Stripe. There’s a lot of nuance in Stripe data. There’s a lot of nuance in Stripe’s business model. Hubert is the guy that sits on top of this Hubble tool, which you use to find, explore, and query Stripe data that does that internally. And it’s early. I mean, we have like 900 people who use it a week. We have tried to focus the people who use it mostly on... Technical folks who know the domain for exactly the reason you were citing earlier, which is it could get the answer fundamentally wrong. And technical folks are going to be better positioned to validate and provide feedback. One of the most interesting things to me as I was going through the Hubert evals was the place where Hubert did the worst was around data discovery. That is to say, it had a hard time finding which table and which field was best to answer the question at hand. And, I mean, personally... As a user, when I know the table, I actually now just articulate the table in the chat interface. But more importantly, we’re doing a big push right now to deprecate low-quality tables and have the owners document high-quality tables. I haven’t yet figured out if I trust an LLM to do the documentation. So, for like the canonical data sets, we’re kind of brute-forcing it with humans who know the domain. The other thing that we are exploring, but we haven’t landed yet, is offline, it looks like there’s actually really... Like, Hubert.com. It does much better if you tell it where in the organization I sit, because it knows, oh, I’m interested in LPM data, or I’m interested in link data, or I’m interested in OCS data. People who work on the optimized checkout suite tend to query these tables, look at these fields, ask these kinds of questions, look at these types of metrics. Now, that’s not in production, but I think there’s this interesting question of, like, we can have humans do some, like, prompt engineering or documentation or whatever. But we can also give the LLM more just historic... We can also give the LLM more just historic context about what people like me liked to do, basically. So, that’s the next step there. I’m bullish on it. I think... I don’t know if it’s going to be two months or two quarters before, like, everyone’s on it. Like, I think it might take some time to get high enough conviction that we’re not going to have an important wrong answer. Text-to-SQL is really easy, though, with really well-structured, well-documented data. It’s just most data is not well-documented and well-structured.

Swyx [01:08:18]: Well, so, before... Immediately before this, I was actually in the data engineering industry. We talked about... And you said you didn’t have an opinion. But I always thought that, like, data discovery is the important victory or, like, the ultimate victory of data catalog people and semantic layer people. Do you agree with those movements in the data world? Do you have any tweaks on the modern data stack that you have?

Emily [01:08:42]: So, we are increasingly moving to, like, semantic events infrastructure. I think the value of near real-time, high-quality... Well-documented data is about to skyrocket because I’m pretty sure that nine months from now, no one is going to want to go and, like, look at a even, like, static dashboard and click around. They’re going to want to be fed insights or they’re going to want their agents to be fed insights and they’re going to want to be able to just, like, pull real-time, high-quality data. For us, what that looks like, like, the two most important domains for our users... In that regard are payments and this, like, usage-based billing, which needs to be very, very real-time for all sort of, like, the AI business model stuff we talked about. And so, there, our path is, like, semantic events, canonical data sets available in near real-time in dashboard, yes, because some people will still use dashboard, in Sigma, so, like, queryable, but also in sort of a Stripe data pipeline type, right? Because very few people want to look at Stripe. They want to look at Stripe data in isolation. They want to look at Stripe data connected to the other stuff, right? So, I mean, you could just even imagine, like, pulling into BigQuery or whatever. They want to see it connected to other stuff. And, you know, historically, honestly, like, it wasn’t all the same data feed for all of those products, and that also creates confusion. So, we’re doing a bit of a re-architecture for that flow starting in the next six months with just billing and payments, and then I think we’ll expand from there. There’s always going to be... There’s always going to be a bunch of data that, for whatever reason, doesn’t fit your, I like to call it a North Star architecture, but, like, your North Star data architecture, right? And I wish that someone could figure out how to make sense of the old bad data so you don’t have to, like, re-architect everything and throw away the old, right? Like, you’ll always have, like, okay, you know, like, there’s, like, the Stripe.com website, which happens, like, you know, before you even create an account. It’s a very different type of data. But, like, you know, how should I reason about that? How should I manage that? And I think traditional enterprises deal with this a lot.

Swyx [01:11:01]: Like converting website analytics to signed-up users and all that? Yeah. Oh, I’ve thought about that so much. Oh, yeah? You just need, like, it’s kind of fingerprinting, which is, like, something that people are kind of uncomfortable with, but you can. You can do it.

Emily [01:11:15]: Yeah, yeah, yeah.

Swyx [01:11:15]: We have, like, a couple questions on, like, I think there’s a build versus buy question. Yep. On you’re building a lot of internal tooling, and that’s great. But also, there’s a lot of great tooling out there. How do you navigate this? Obviously, you have a lot of unique internal context, but you also have, you work with external vendors. Like, people, I guess, want to know how to work with you, but also people in your shoes at peer companies also want to know how you do this decision. Yeah.

Emily [01:11:40]: I think for us, it’s not an either or. It’s very much an and when it comes to build versus buy. And some of that and is sequential, right? So, you and I were talking earlier. When GPT 3.5, I think, like, first hit the scene, we’re like, oh, everybody at Stripe needs to have access to LLMs, but, like, we don’t quite know how to do that in a way that’s, like, enterprise-grade safe and we feel good about. Like, we don’t see a provider there right now, and so we built it. But now we use, like, open source, LibraChat, you know. So, I think there’s, like, there’s, like, an evolution over time. And one of the things that I think can be really hard, especially for the team who has tunnel vision for the products they own, you know, you love your product, you want to make it better over time. Yeah. Because you can get stuck in a lot of hill climbing. Like, we could have taken Go LLM and been like, oh, we should figure out a way to, like, give it access to Toolshed. Oh, we should figure out a way to make it do, like, deep research or, you know, like, we could have done that. And sometimes you just need, like, sort of more of an outside-in perspective of, hey, if I ignore the sunk cost fallacy, ignore my emotional connection to the thing that I spent nights and weekends building, first principles, like, if I were to do this today, what would I do? And some of that is also making sure people. A lot of confidence and conviction in their own abilities and the fact that there’s a ton that they can contribute to the company across domains to kind of liberate them from needing to own this product. Another thing that we’ve done, and this is especially true in the AI space just because there’s, like, so much new stuff coming online. We call it the spotlight program. But basically, like, we put out RFPs for products that we want to buy. Ooh. So one example that we did recently. I guess it was, like, a year ago now was evals. So we were, like, okay, here’s our problem with evals. Like, who’s going to solve it? And we put out this RFP and the spotlight program. We obviously see a lot of these companies directly because they run on Stripe and or, like, their investors have some affiliation with Stripe. And so we know them. So we actually had, like, more than two dozen applicants for this evals RFP. There’s no way you can evaluate all of them. Well, we actually, we did. Oh, my God. They wrote, like, nice one-pagers. We read them all. We narrowed it down to two finalists. Braintrust. Braintrust ended up winning. We did a POC with them. We’ve had them on, yeah. They rock. We stuck with them. We also love, like, Weights and Biases, Flight, Kubernetes. Like, sort of, like, sort of, like, the basic stack you can think of. But there have also been cases where we have had to build. So one example is, if you think about traditional ML for a second, also relevant in the world of, like, our foundation models and that embeddings basically, like, can become features, our feature engineering platform. So we had a homegrown feature engineering platform. It was old. It was on its last legs. We had a team internally that really wanted to adopt Tekton. We evaluated Tekton. This was a couple years ago now. At the time, we couldn’t wrap our heads around using it on the charge path, just from, like, a latency and reliability perspective. Like, we’ve got to be operating at six nines. We’ve got to be, like, decisioning in, you know, tens of milliseconds for some of these models. Like, we just, we can’t reason about that. We can’t reason about that on the charge path. We ended up pairing up with Airbnb and building, we call it Shepard internally, but it’s open source under the name Cronon. You know, I think there are cases for both. But we always start with, like, what could we buy? Sometimes those are obvious solutions. Then we say, oh, there’s no obvious buy solution, but, like, maybe there’s some new startup thing. Let’s run the Spotlight program. And if we really come up dry, then we will build. And as we build, we reason about, okay, six months from now. Twelve months from now, should we still be building or should we actually swap in a buy solution because the market has evolved? The other thing that we feel pretty strongly about in the world of AI is there will be many model providers. There will be many models. And we do not want to hitch our wagons to just one horse. You know, there’s lots of enterprise-grade versions of choose your LLM provider. Like, that’s of much less interest to us. Um, then solutions that sit on top of many different providers and many different models and allow us to swap in and out. Yeah.

Alessio [01:15:57]: And this is the Stripe experiments team. When you decide something, we need to go. You have the same people always do this.

Emily [01:16:04]: Or do you rebuild this team based on if it’s evals or if it’s like, it’s probably a bottoms up, like whoever it’s bottoms up. So experimentation happens in a lot of different places. So like for the evals, we just did it. Like we ran it within ML infra with like the new rebuild of GoChat. We paired up. We paired up someone from EP with the people who had been owning, um, go LLM, you know, one of the things that’s interesting about experimental projects is the goal is to learn quickly, whether there’s product market fit, whether that’s with your internal users or your external users. But the goal is to basically get to escape velocity, like have a product that launches and goes live, whether that’s like GoChat GA and replacing go LLM or whether that’s like token billing, uh, serving our users or agentic commerce now being a thing. And what we found. And, you know, this is the experimental projects team has been around, um, about a year and a half. And so these are relatively small samples we’re talking about, but what we have found in that sort of Anik data is what we call embedded projects, projects where you take a couple people from a product or infrastructure team and a couple people from the experimental projects and group them together are more likely to reach escape velocity. Token billing is a good example, right? Like we need the billing team to take it forward. And if the billing team was like core to it from the start, it’s much easier for them to take it forward. Same thing, you know, if, if the ML infra team, you know, deeply understands the new build and feels ownership over it, it’ll be successful, um, in the long run. So I don’t think of that, you know, some people think of these teams as like labs teams off in the side, off in a corner operating totally in isolation. We do some of those, like we’re doing that for agent pay, like agent to agent payments. Cause that, that just like needs a big, rev on like what’s the product shape, what’s the technology shape, but wherever possible, we actually do it as a joint project, very deeply embedded with design partners, like with customers that want to do it with us, but also with, uh, with other teams that stripe.

Swyx [01:18:01]: My typical line on just, just closing the loop on the build versus buy thing, uh, is usually buy then build. If you think about the sequencing, I think yours is, yours is much more nuanced in terms of like.

Emily [01:18:12]: Buy then build if a buy exists. Correct. If a buy doesn’t exist, you might want to build, but pick up your head of recording. To make sure you can’t buy. Yeah. Yeah.

Swyx [01:18:20]: Well, and also I think like mostly because I, I see the opposite when P a lot of people try to do the opposite of build then buy to, to, to like reason things from first principles, but actually, uh, the sheer amount of experimentation in the wider world means that a lot of people are being specialists in your thing, like devils, where you can just benefit from that, from their experience instead of, uh, reinventing the wheel.

Emily [01:18:43]: So before Stripe, I was at Coursera. Have you guys ever looked like the ed tech platform? Okay. Of course. I was there for eight years and I joined when we were less than 40 people and it was a lot of absolutely brilliant folks from Andrew and Daphne’s lab at Stanford who had never had a job before. And by the way, I do not count myself in the absolutely brilliant folks from Andrew and Daphne’s lab. I was on the east coast, um, and not absolutely brilliant, but I had also never had a job before. And a bunch of us never had a job before, but like really hardworking, determined people built a bunch of stuff homegrown that we shouldn’t have. Right. Yeah. We had our own experiment. We had our own experimentation platform. We had our own analytics platform. We had our own machine learning platform. We had our, uh, we learned a lot. We learned a lot. But like, what is Coursera’s core competitive advantage? It’s not their experimentation platform. Um, and you know, that was 2014. So actually a bunch of that stuff didn’t exist, but it was painful in 2018, 2019, 2020 to rip and replace and rip and replace was definitely the right thing to do. The other sort of, um, thing I’ll note on that is if you look at how Stripe is the skeletal system. I mean, if you look at the AI system for all the AI companies, it very quickly becomes clear that when it comes to payments, billing, tax revenue recognition, reporting fraud protection, consumer checkout experience, pricing and monetization frameworks, they are completely by like, they’re completely buying Stripe. That’s all they’re buying. Um, and I think there’s an interesting thing there where I was talking to an AI company the other day who uses another provider. Yeah. block bots at the time of signup. And they said, it’s actually really annoying to have multiple third parties doing my fraud protection, like one doing it up funnel and the other doing it down funnel. Well, they asked to switch to Stripe. We don’t yet have that particular functionality, but we could build it. But I think there’s also, it’s also important to reason about what is the third party that you can be, not for everything, but like more all in on so that it plays nice internally so that you have fewer relationships so that you have like preferential pricing, et cetera, et cetera. And you know, when you think about cloud providers, like that often

Swyx [01:20:50]: happens a lot as well. Vercel is definitely doing that. Totally. Trying to bundle. Yeah. So this is the economy section. We were saying you’re more interested in sort of the AI economy

Emily [01:20:59]: takes. The obvious big one is, are we in a bubble? Are we in a bubble? Okay. So it depends what you

Swyx [01:21:04]: mean by a bubble, but I think, you know, one question. Classic economist answer is like,

Emily [01:21:10]: it depends. It depends. I know, There’s always, there’s like two armed economists on the one hand, on the other hand. Okay. Okay. Okay. Okay. The question I got a lot a couple of quarters ago, especially because all of these AI companies are private, is are they creating real value? Is there real revenue coming in? Right? Because it’s pretty clear to see that there are real costs. There’s very big fundraisers. There’s a lot of capital that’s flowing out. Is there dollars flowing in? And so that actually forced us to

Swyx [01:21:41]: step back. You know, one of the fun things at Stripe is just like, you just see it going through,

Emily [01:21:45]: right? You see each successive wave of startups. You see, you know, people retaining and churning their subscriptions. You see who’s buying what for how much from whom. And when we stepped back and we said, okay, like, let’s just look at this AI cohort. And there’s lots of different ways to define it. But for simplicity, one cohort that we looked at was the 100 highest grossing AI companies on Stripe. And you kind of need a reference point. And so we were like, let’s compare them to the 100 highest grossing SaaS companies from five years. And we looked at things like, how quickly do they get to a million or 10 million or 30 million in ARR? And the answer is two to three times as fast as a SaaS cohort. Yeah. We looked at questions like, how diversified, global is their customer base? And the answer is, at the end of their first year, at the end of their second year, basically, whenever you look, they are twice as global. Like, they’re selling into twice as many countries. 55 countries. They have more. Majority of their revenue coming from outside their home market, even if their home market is the U.S. And in some cases, you know, there’s a startup in France who’s in that list, who’s like 95% of their revenue is outside of France, right? They’re very global. And then you start to look at things like retention, which also comes up a bunch, right? Like, is this truly ARR or is this

Swyx [01:23:07]: like revenue popping and then falls off? Times 12.

Emily [01:23:11]: Exactly. And this one was a little more nuanced. So, if you squint at the data, you can actually see that these AI companies on a per-company basis have slightly lower retention than the SaaS companies. Not, like, dramatically lower, but slightly lower. And that’s also consistent with being, like, relatively early in the adoption curve, but even correcting for that, slightly lower. But then if you bundle that, if you look at, okay, like, these SaaS companies are doing this wave of things. These AI companies are doing this wave of things. What’s interesting about SaaS is the churn is churn from the entire vertical. In AI, They’re just churning from that company and flipping to another company. And then if you keep watching them, a few months later, they flip back to the first company. So that tells me it’s actually like a very competitive market. People like the product. They want to use the product. But there’s a bunch of good products. And the best product is changing over time. And so people are happily flipping across. Oh, well, so in SaaS, you had this cool trend of you started horizontal. You started with Salesforce. And then you went vertical, like you have the vertical SaaS, the toasts and the whatever else. We talked a bit about wrappers earlier. AI has done the same thing. You start horizontal, right? You’re like the infrastructure. You’re the model providers. You’re the purely horizontal. And then all of a sudden, it’s like all of these verticals. It’s like, OK, we’re in healthcare. And there’s Nabla. And there’s Ascribe. Or we’re in architecture. And there’s Studio. Or we’re in law. And there’s Harvey. Just like all of these verticals popping up. And popping up much faster than in SaaS. And I think there’s two things happening there. One is you can get to those verticals very efficiently because you’re sitting on top of someone else’s LLMs. You don’t actually have to do that. You don’t have to do the research. And it’s like a quite lightweight build. But the other thing is, because these AI solutions are so borderless, niche markets, vertical markets, at a global scale, are actually quite large markets. And so there are incentives to specialization in a way that maybe there weren’t five years ago. So anyway, is it a bubble? I don’t know. It depends how you define a bubble. I’m a two-armed economist. But what I will tell you is. Because these are companies that are growing very quickly, faster than anything we’ve ever seen, very diversified in their customer base, which makes me feel better about them. Very sticky in their customer base, not always on a per company level, but on like a problem to be solved level, which tells me that the customers are getting recurring value from the product and are willing to pay for it. Yeah.

Swyx [01:25:34]: What I’m hearing is like, there is some real, like better quality businesses being built at the same time that it has no, the expectations can raise. And that’s not within the Stripe observable universe. Yeah.

Emily [01:25:47]: The part we didn’t, I mean, the part we didn’t talk about was the cost profiles and, you know, when I reason about cost profiles, there’s really like two, there’s like the fixed costs and the marginal costs, or there’s like the people costs and like the, I mean, in the case of AI, like the inference costs. Right. And so the people costs for these AI companies are quite small, right? You look at a lovable, you’re talking crazy revenue milestones. Right. With 10, 20, 30, 40 people in the early days. And even today, right, when you, when you look at most of the top 100 AI companies on Stripe, their revenue per employee is unlike any other business, including public companies who are known for being incredibly efficient companies. That of course, people cost ignores the inference costs. And so I think we absolutely need to model assumptions around the efficiency and where the inference costs is going in order to be able to. But in the same way we were talking about, how do we think about the ROI on, on these coding agents? I think we would be unwise to measure the value of these companies under the assumption of today’s costs. And we need to model out based on, you know, reasonable things that we’ve seen and reasonable expectations we have about the world, like those costs going down quite a bit, at which point, you know, in traditional senses, like very interesting businesses. Yeah.

Swyx [01:27:08]: I would say like, you know, there’s the benign element and then there’s the less benign in terms of the cost profile, which is yes, as, as AI is increasingly doing more and more labor that you, you would otherwise have hired for, then it should rise as, as a part of your spend. And then there’s the less benign one, which is like people are selling dollars for 50 cents. And that’s why you’re, you’re seeing such revenue traction because obviously you’re kind of giving money away.

Emily [01:27:32]: For sure. And, you know, like I, I was a grad student in the early days of Uber and DoorDash or, you know, I was year one. And two of Coursera, which was like, that’s still a.org at that time, right? Like basically a nonprofit. And I remember my lifestyle was subsidized by the VCs who were paying for part of my Uber and paying for part of my DoorDash. So, you know, we’ve lived that pain. Those two worked out. Some of those prices going up. But increasingly what we’re seeing from the AI companies on Stripe is they do want to have healthy unit economics. I mean, let’s not talk about like the big labs that are pouring crazy money into research. But if you’re talking about like the vertical kind of wrappers, which are themselves also doing, you know, like, you know, like, you know, like, you know, like, you know, like, you know, like, you know, like, you know, like, you know, like, you know, like, you know, like, very well as businesses, they are building quite healthy unit economics. And the demand we’ve seen for token billing, I think, is actually in part a testament to the fact that they really do want to have unit economics. So not their overall book, but literally like the marginal person I serve, the revenue I get from them versus the cost I incur for them, they want those to already be in the green. So I think there are some some very good businesses that are being built.

Swyx [01:28:37]: We’ve kept you for a long time. You’ve indulged us in so many different topics. Do you have any other hot takes on AI in the economy that you want to indulge in? My classic hot take is how come AI doesn’t show up in the GDP per capita numbers, which is part of the whole productivity discussion, but it’s really, really driving home. We have to see this show up somehow, and that’s part of the bubble discussion. But I think to me, any story where technology as a factor in the macroeconomy equation is supposed to be a big driver, you should see it in GDP at some point. GDP doesn’t measure everything, but at some point.

Emily [01:29:16]: We should see it in GDP. There’s a lot of noise in GDP because there’s a lot of other drivers of GDP.

Swyx [01:29:21]: How quickly we see it is, I think, an open question. It should be fast. It should be fast-ish. The funny hot take is the only way we’re seeing GDP is the data center build-out. Oh, interesting.

Emily [01:29:34]: Oh, yeah. I’m less close to that. It could be. I mean, hot takes. I think AI should make markets more efficient. I think agents should make commerce more efficient, which should genuinely expand the aperture of what people buy. I think agents are already making business creation more efficient, which should accelerate new startup growth, which we are also seeing. And, you know, if you just think about, like, the tens of thousands, hundreds of thousands of businesses that are getting started in these AI dev tools that, like, would not have been getting started before, I think that’s incredibly promising. I think we have a real cost question on our hands, but I think it will be solved. For many domains, I think there will be cheap enough models to do the job that create meaningful value. I think the AI... Like, companies are being quite savvy. We talked a bit about the unit economics, but also, like, what are they pricing to, right? So, SaaS was mostly seat-based, and you could imagine sort of a death spiral where, like, AI is seat-based, but AI is replacing the seats, and so you need fewer seats, and you monetize less. Like, you don’t want to peg your revenue to the thing you’re trying to replace, right? And I think, you know, outcome-based or usage-based will be... Much more powerful. I’m seeing more adoption of AI outside of the U.S. in a broad range of markets than I expected.

Swyx [01:31:11]: Brazil is huge. Brazil is huge. Yeah.

Emily [01:31:13]: It’s a little bit hard to parse what is, like, well, Stripe is opening up more markets and having more payment methods and giving them more exposure versus, like, literally there is an expansion in adoption. But I think it will be promising for the world if there is more... It’s not really a quality of access. It’s not really a quality of access because, like, on paper, anyone has access, but, like, equality of adoption so that we don’t lead to sort of... We don’t end up with very uneven economic growth as a result of AI. So, we’ll have to watch. I don’t think it’s going to show up next year. I don’t think most businesses are targeting employee efficiencies next year, but I think every business is targeting employee efficiencies for 27 and for 28. Yeah. Which is suggesting more efficiency. And if you can couple more efficient production with more efficient consumption, which, frankly, is what agents do, then one would expect, yes, GDP to rise, to rise meaningfully. And I think you could debate, is it one percentage point more growth per year? Is it three percentage points more growth per year? Like, I don’t know. I don’t think it’s 10. I don’t... I mean, I’d love it, but I don’t think so. I think we’ll have to see. Because that thing compounds, right? That thing compounds. That thing compounds.

Swyx [01:32:32]: GDP is a big number. Yeah. But yeah, the term I’ve had for the movement of employee efficiency is tiny teams. Like, you know, teams with more millions in revenue than employees, which, like, completely changes the startup structure because you are probably profitable from, like, maybe your first round of funding. Yeah.

Emily [01:32:51]: I have one more hot take, which is it’s easy to think in a world of, like, really exciting, powerful tech that somehow brand doesn’t matter. It’s all about the technology. But actually, if you look over the last year, so much of the, you know, value created in AI companies has actually come from... And Lovable was brilliant in branding themselves Lovable, right? Like, so many of these rappers are actually winning on a really differentiated user experience and a really compelling brand and a really compelling community. And so, I don’t know. I just... You know, sometimes, you know, investors, friends, whatever, will be like, like, oh, like, what do you think of this? What do you think of that? And it’s like, you know, great founders who are highly technical are amazing, but you also need them to be hyper-focused on, like, the user and the product experience and really creating something, like, beautiful and craftsy. And I think some people are like, oh, like, AI should replace that. And, like, all that matters is the tech. And there’s not going to be a human internet. And it’s just going to be agents talking to agents. And it’s like, I don’t know, maybe, but, like, so far, what I’m seeing is brand matters more than ever. Yeah.

Swyx [01:33:57]: The Silicon Valley phrase is you need Riz and Tiz. And if you don’t have Riz, then it’s awful. I think, like, Stripe, something Stripe has always embodied, like, very good technologies with also industry-leading design, which I think is very important.

Emily [01:34:11]: I’m a Katie Dill fangirl. Katie Dill’s our head of design. Oh, okay. Actually, I was co-hosting Friday Fireside, which is, like, our weekly company thing today. And I cited something that Katie’s team did and made it clear that I was a fangirl. And Tanya, who’s our head of PMM, said she was going to have to fight me for, like, Katie’s biggest fan. Anyway, we decided that the three of us would just go to a spa and drink mimosas instead of fighting. But for a hot second in the company’s Slack chat, there was, like, maybe a fight between TK and I.

Swyx [01:34:42]: Okay, let me, while you’re on this topic, what’s something that you learned from her that, like, has really driven design at Stripe? Yeah.

Emily [01:34:49]: Katie does not give an inch on quality. And it doesn’t matter if it’s, like, one banner that 2,000 users see. That, you know, has some font that’s slightly bigger than it should be. Like, that’s a bug. That has an SLA. That needs to be burned down. And actually, now every two weeks we have a run the business review where it’s, like, a 60 folks get together and talk about the whole business. And literally each of us has a slide that’s, like, did we meet our bug burned down SLA for these, like, largely, not exclusively quality, but often quality issues. And I think she builds, like, beautiful experience. She and her team design, like, beautiful experiences and sort of, like, macro are, like, extremely innovative. But there’s something that I’ve learned around, like, the micro. Like, you have to obsess over every detail. And one tiny thing that’s not good enough is worth all of us sweating until it is good enough. It can be a little exhausting at the scale of Stripe, but it’s also, like, very grounding. Yeah. To just know, like, there’s a clear line. And if it doesn’t meet the quality bar, like, you just got to fix it.

Swyx [01:36:07]: Wow. Well, thank you for spending some time with us and explaining how things work at Stripe. I mean, like, everyone’s always curious and you’ve been very generous with your time.

Emily [01:36:16]: Oh, thanks for having me. And it’s been really fun.

Swyx [01:36:18]: Call to action. Hiring, I assume? Yes.

Emily [01:36:21]: We are hiring. We are hiring. I mean, we’re hiring everybody. But we are particularly hiring machine learning engineers. Slash scientists. A lot of back-end folks. So, like, if you’re excited to, you know, build the infrastructure for building agents or build the infrastructure to do machine learning, a lot of those. We are recognizing that data is, like, increasingly an asset that our users want real-time and high-quality and well-documented. So, like, if you’re big on data engineering or building data platforms, also hiring there. But just, like, across the stack. It’s, like, a great team and fun project. So, yeah.

Alessio [01:36:59]: We’re hiring. Thanks, Emily. Awesome.

Emily [01:37:01]: Thanks, folks.

Apart from the Jony Ive and Stargate stories happening offstage, the biggest takeaway from OpenAI’s 2025 Dev Day was how the company is commited to being a proper developer and app distribution platform even as it scales a leading frontier research lab, consumer chatbot, and new social network. For a developer, the number of product names can be a little overwhelming, so we drew you a little flow chart:

As the only podcast to deeply cover DevDay 2023 and DevDay 2024, we were very impressed by DevDay 2025 and consider it the best one yet, both in terms of on-the-ground execution and the confidence expressed in the launches. Even if you don’t have time to read through the full slate of releases, the 50 min keynote is a must watch:

One of our most popular episodes last year was with Michelle Pokrass, then of the API team, who deeply believed that APIs are the solution for developers to do the last mile of AGI distribution with OpenAI, even if AGI isn’t fully here yet, aka “API = AGI”:

And it seems like OpenAI DevDays are the company’s commitment to that mission and a thriving developer ecosystem (for those who manage to not get steamrolled).

But first, let’s get the “smaller” launches out of the way.

One very impressive thing about OpenAI is that they continue to not hold models back in the API. Every year there has been some excitement about a ChatGPT capability leading into a DevDay, and every year OpenAI has not disappointed by holding back some capability from API release: DevDay 2023 included GPT4 Vision, Dall-E 3, and Code Interpreter, Dev Day 2024 gave us the Realtime Voice API while everyone was still excited about it.

Dev Day 2025 gave us every frontier OpenAI API we wanted:

• gpt-5 pro (model): https://platform.openai.com/docs/models/gpt-5-pro

• sora-2 (model): https://platform.openai.com/docs/models/sora-2

• sora-2-pro (model): https://platform.openai.com/docs/models/sora-2-pro

  • Video generation with Sora (docs): https://platform.openai.com/docs/guides/video-generation

  • Sora 2: Prompting Guide (cookbook): https://github.com/openai/openai-cookbook/blob/16686d05abf16db88aef8815ebde5c46c9a1282a/examples/sora/sora2_prompting_guide.ipynb#L7

and then some distillations we didn’t even know we wanted:

• gpt-realtime-mini-2025-10-06 (model): https://platform.openai.com/docs/models/gpt-realtime-mini (70% cheaper)

• gpt-audio-mini-2025-10-06 (model): https://platform.openai.com/docs/models/gpt-audio-mini

• gpt-image-1-mini (model): https://platform.openai.com/docs/models/gpt-image-1-mini (80% cheaper)

Prompting some somewhat nice comments:

“Today we’re going to open up ChatGPT for developers to build real apps inside of ChatGPT. This will enable a new generation of apps that are interactive, adaptive, and personalized that you can chat with. To build them, we are launching a new Apps SDK, now available in preview. With the Apps SDK, you get the full stack: you can connect your data, trigger actions, and render a fully interactive UI and more. The Apps SDK is built on MCP. You have full control over your backend logic and frontend UI. We’ve published the standard so anyone can integrate the Apps SDK. When you build with the Apps SDK, your apps can reach hundreds of millions of ChatGPT users.” — Sam Altman

When I first wrote Why MCP Won back in March, I didn’t expect OpenAI to adopt it so quickly, much less to now be the first to turn it into the basis of a full app store platform. I’ve been guilty before of getting too excited about ChatGPT’s attempts to be an app store, but objectively this is the best shot yet (though success is far from guaranteed). I happened to be listening to an entrepreneurship podcast telling the story of starting a $745m business the day Mark Zuckerberg launched the Facebook developer platform in 2007, and it was an interesting reminder that while the Apps SDK itself may not be the most exciting to all developers, it is very very exciting for the kind of developer that knows how to take advantage of a new app store to get abnormal distribution. In his media briefing yesterday, Greg Brockman even said that some people were able to build sustainable businesses off of Custom GPTs — while certainly not a common experience, a more full fledged App Store certainly creates more opportunity.

I also think the technical accomplishments in making this happen should be appreciated better - from the innovative React component bundling for an iframe target solution so you can provide -any- custom UI component to your ChatGPT App, to the live data flow demonstrated by the Coursera App in the keynote, where live video state is conveyed to ChatGPT so you can do arbitrary queries as you watch.

First of all, the box set at Dev Day was hilarious:

But there is a LOT to unpack here and you can be easily forgiven for missing some pieces:

This is where the bulk of today’s podcast lies, as we got to sit down with Christina after her big Ben Thompson approved demo and ask her and Sherwin more about the finer points of the Agent Builder and broader AgentKit vision:

As a side note, we didn’t cover this in the pod but it is interesting that Guardrails was one of the most exciting open source AI Engineering projects 2 years ago, but since Nvidia joined the fray and now increasingly more and more developer platforms are introducing their own guardrails, they seem to be another 2023-era idea that was more of a feature than a product, like vector indexes for embeddings (Shreya has since gone up the stack to Snowglobe, a simulation engine for user testing AI products, check it out).

• Sherwin Wu: Head of Engineering, OpenAI Platform https://www.linkedin.com/in/sherwinwu1/ https://x.com/sherwinwu?lang=en

• Christina Huang: Platform Experience, OpenAI https://x.com/christinaahuang https://www.linkedin.com/in/christinaahuang/

Thanks very much to Lindsay and Shaokyi for helping us set up this great deepdive into the new DevDay launches!

Alessio [00:00:04]: Hey, everyone. Welcome to the Latent Space podcast. This is Alessio from the Recurrent Labs, and I’m joined by Swyx, editor of Latent Space.

swyx [00:00:11]: Hello, hello. And we are here in the OpenAI Dev Day studio with Sherwin and Christina from the OpenAI platform team. Welcome. Thank you for having us. Yeah, it’s always good to be here. Yeah, it’s such a nice thing. We’ve covered like three of these Dev Days now, and this is like the first time it’s been like so well organized that we have our own little studio, podcast studio, in the Dev Day venue. And it’s really nice to actually get a chance to sit down with you guys. So thanks for taking the time.

Sherwin [00:00:38]: Yeah, I feel like Dev Day is always a process, and we’ve only had three of them, and we try to improve it every time. And I actually, I know for a fact that I think we have this podcast studio this time because the podcast interviews and the interviews with folks like yourselves last time went really well. And so I want to lean into a little bit more. I’m glad that we were able to have this studio for you all. We were kneeling on the ground interviewing like Michelle last year. I don’t know.

Alessio [00:01:00]: I just saw it post-production. I thought it was...

swyx [00:01:03]: We had to have people like... We had to like cordon off the area so they wouldn’t walk in front of the cameras.

Alessio [00:01:07]: Yeah, people would just come up, hey, good to... I’m like, we’re like recording.

Christina [00:01:10]: I guess if you guys have been to three, like what stood out from today or what’s your favorite part?

swyx [00:01:17]: I feel like the vibes are just a lot more confident. Like you are obviously doing very well. You have the numbers to show it. You know, I just... Every year in Dev Day, you report the number of developers. This year it’s 4 million. I think last year it was like three. And I have more questions. I’ve got that kind of stuff. But also like just like very interesting, very high confidence launches. And then also like I think the community is clearly much more developed. Like I think there’s just a lot more things to dive into across the API surface area of OpenAI than I think last year in my mind.

Alessio [00:01:56]: I don’t know about you. Yeah, and we were at the OG Dev Day, which was the DALI hack night at OpenAI in 2022. And I think Sam spoke to like 30 people. So I think it’s just crazy to see the...

Sherwin [00:02:06]: Yeah, honestly, I think it’s like it’s kind of similar to this podcast studio, which is I think we’ve had a number of Dev Days now. We honestly were like slowly figuring things out as a company over time as well. And both from a product perspective and also from a like how we want to present ourselves with Dev Day. And at this point, we’ve had a lot of feedback from people. I actually think a lot of attendees will get like an email with like a chance for feedback as well. And we actually like do read those and we act on those. And like one of the things that we did this year that I really liked were all of those. Like there was like some art installations and like the little arcade games that we did, which was, you know, came up with via like engaging with the feedback.

Christina [00:02:43]: Yeah, the arcade games were so fun. I loved like the theme of all the ASCII art throughout. This is my first SF Dev Day, but I’ve been to the Singapore one. That was actually my first week there. Oh, yeah, that’s the one I spoke about. Yeah, I saw you there. That was my first week of OpenAI. So really in the deep end.

swyx [00:02:57]: Put it on a plane to Singapore. Yeah. Yeah, that’s awesome. Well, so, you know, that’s congrats on everything. And like kudos to the organizing team. We should talk about some developer API stuff. Yeah. So we’re going to cover a few of the things. You’re not exactly working on Apps SDK, but I guess what should people just generically take away? What should developers take away from the Apps SDK launch? Like how do you internally view it?

Sherwin [00:03:22]: So the way that I think about it is I actually view OpenAI since the very beginning as a company that is really valued. Kind of like opening up our technology and like bringing it out to the rest of the world. One thing we talk about a lot internally is. You know, our mission at OpenAI is to, one, build AGI, which we’re trying to do, and then, but two, you know, potentially, you know, just as important is to bring the benefits of that to the entire world. And one thing that we realized very early on is that we as a company, it’s very difficult for us to just bring it to every truly every corner of the world. And we really need to rely on developers, other third parties to be able to do this, which is, you know, Greg talked about the start of the API and like kind of how that was formulated. But that was part of, you know, that mentality, which is we. We need to rely on developers and we need to open up our technology to the rest of the world so that they can partake for us to really fulfill our mission. So the API obviously is a very natural way of doing that, where we just literally expose API endpoints or expose tools for people to build things. But now that we have, you know, ChatGPT with its 800 million weekly active users, I forgot the stat that we shared. I think it’s like now the fifth or sixth largest website in the world.

swyx [00:04:30]: And the number one and number two most downloaded on the Apple App Store.

Sherwin [00:04:35]: Oh, yeah, with Sora, yeah, yeah, but I don’t like it moves around all the time.

swyx [00:04:37]: So it’s kind of hard to celebrate and, you know, just screenshot it when it’s good.

Sherwin [00:04:40]: Yeah, we definitely screenshot it when it was good. But kind of going back to my main point is like we’ve always kind of engaged the developers as a way for us to bring the benefits of of AGI to the rest of the world. And so I view this as actually a natural extension of this. Candidly, we’ve actually been trying to do this, you know, a couple of times with the last dev day with GPT, two dev days ago with I’m sorry, two devs ago. With GPTs and plugins, which was, I think, not tied to a dev day. So I view this as like, again, we love to deploy things so iteratively and I view it as like just a continuation of that process and also engaging deeply with developers and helping them benefit from some of the stuff that we have, which which in this case is ChatGPT distribution.

Alessio [00:05:23]: And when so apps SDK is built on the MCP protocol, when did OpenAI become MCP built? I’m sure internally you must have had, you know, the signed discussions before about doing your own protocol. When did you buy into it and how long ago was that?

Christina [00:05:38]: I think it was in in March, I want to say. It’s hard for me to remember kind of like the exact.

swyx [00:05:42]: March was the takeoff of MCP.

Christina [00:05:44]: Yeah, yeah. So we built the agents SDK and we launched that alongside the responses API in early March. And I think as MCP was growing, that felt like a really and, you know, we’re building kind of a new agentic API that can call tools and just be much more powerful. MCP was kind of like the natural protocol that developers were already using.

Christina [00:06:05]: I think like in March is when we added in MCP to agents SDK first and then soon after with kind of our other.

Sherwin [00:06:11]: Yeah, I think there was like a tweet or something we did where it was like OpenAI, you know, is.

Christina [00:06:15]: Yeah, there was definitely a moment. I think there was a specific moment in a specific tweet.

Sherwin [00:06:19]: But what I will say, though, is like, and this is honestly the credit to the team at Anthropic that kind of created MCP is I really do think they treat it as an open protocol. Like we work very closely with, I think, like David and the folks on the like, you know, consortium. And they are. They are not, you know, really viewing it as this like thing that is specific to Anthropic. They really view it as this open protocol. There is like it is an open protocol. The way in which you make changes feels very open. We actually have a member of our team, Nick Cooper, who is sitting on kind of like that, that steering committee for MCP as well. And so I think they are really treating it as something that is easy for us and other companies and everyone else to embrace, which I think they should because they they do want it to be something that is very embraced by all. And so because of that, I think it makes it a little bit easier for us to. To embrace it. And honestly, it’s a great it’s a great protocol. Like, it’s very general. It’s already solved. Like, why would you make it? Yeah, yeah, it’s very general. There’s obviously still more to do with it, but it was very easy for us to, you know, integrate because of how how how streamlined and how simple it was. Yeah.

swyx [00:07:18]: My final comment on apps SDK stuff and then we’ll move to AgentKit is, you know, like I always see like in abstractly when you sort of wireframe a website or an AI app, it used to be that the initial AI integration on the website would be you have to have an API. You have the normal website and then you have a little chat bot app and now it’s kind of like inverted where there’s ChatGPT at the at the top layer and then it’s like turn out the website embedded inside of it. And it’s kind of like that inversion that I honestly have been looking for for a little bit. And I think it’s really well done, like actually all like the integrations and the custom UI components that come up. You had like Canva on the keynote there and it looks like Canva, but like you can chat with it in all your the context of your ChatGPT. That is an experience I’ve never seen. Yeah.

Sherwin [00:08:06]: And I think that that’s kind of back to the iterative like learning that we’ve had that I think was because we’ve learned a lot from plugins. So like when we launched plugins, I remember one of the feedback that we got, I don’t know if, you know, people here really remember plugins. It was like March. Oh, yeah. Twenty three. Yeah. I’m like one of the points of feedback was like, oh, you can integrate retail. We told like, you know, all these companies that you can integrate these plugins into ChatGPT, but they really didn’t have that much control over how exactly it was used. It was really just like a tool that the model could call and you were just like really bound by ChatGPT. And so I think like you can kind of see the evolution of our product with this. And like this time we realized how important it was for companies, for third party developers to really own and like steer the experience and make it feel like themselves, help them, you know, like really preserve their own brand. And so and, you know, I actually don’t think we would have gotten that learning had we not, you know, had all these other steps beforehand.

Alessio [00:08:54]: Awesome. Christina, you were the star today on stage with the agent kit demo. You had eight minutes to build an agent. You had a minute to spare. And then you.

Christina [00:09:03]: Yeah, I wasn’t sure, honestly, I was like, let’s do a little bit less testing and maybe we, I don’t know how much time I killed on that. I was extremely stressed when the download came in.

Sherwin [00:09:14]: I was like, if a UI bug is what like takes the demo down, I’m going to be so sad.

Christina [00:09:17]: I think it was a full screen, yeah, like focus thing. I heard the window wasn’t in focus or something, yeah.

Alessio [00:09:23]: Maybe you want to introduce agent kit to the audience.

Christina [00:09:26]: Yeah, so we launched agent kit today, full set of solutions to build, deploy and optimize agents. I think a lot of this comes from working with API customers and realizing how hard it actually is to take, to build agents and then actually take them into production. Hard to get kind of that confidence and the iterative loop and writing prompts, optimizing them, writing evals, all takes a lot of expertise. And so kind of taking those learnings and packaging them into a set of tools that makes it a lot easier and kind of intuitive to know what you need to do. And so there’s a few different building blocks. That can be used independently, but they’re kind of stronger together because you then get the whole end to end system and releasing that today for people to try out and see what they build.

swyx [00:10:13]: Yeah, so I find it hard to hold all the building blocks in my head, but actually chronologically, it’s really interesting that you guys started out with the agent SDK first, and then you have agent builder, you have a connector registry, you have chat kit, and then you have the eval stuff. Am I missing any major components?

Sherwin [00:10:32]: Those are the main moving parts, right? Yeah, I think that’s it. And then, I mean, we also still have the RFT fine tuning API, but we technically group it outside of the agent kit umbrella.

swyx [00:10:42]: Got it, got it, got it. Yeah, so it’s weird how it develops, and it’s now become the full agent platform, right? And I think one thing that I wasn’t clear about when I was looking at the demo was, it’s very funny because what you did on stage was build a live chat app for Dev Day’s website.

Christina [00:11:02]: Yeah, did you get a chance to try it out?

swyx [00:11:03]: Yeah, I tried to try it out. It was awesome. And actually, I kind of wanted to ask how to deploy it. Where’s the merch? Yeah, exactly. I was like, where’d you click the merch? Anyway, and this is very close to home because I’ve done it for my conferences, and it’s a very similar process. But I think what was not obvious is how much is going to be done inside of agent builder. I see there are some actually very interesting nodes that you didn’t get to talk about on stage, like user approval. That’s a whole thing. And like transform and set state, like there’s kind of like a Turing complete machine in here.

Christina [00:11:35]: Yeah. Yeah, so I mean, I think again, like this is the first time that we’re showing agent builder, and so it’s definitely the beginning of what we’re building. And human approvals, like one of those use cases that we want to go pretty deep on, I think. The node today that I showed is pretty simple, like binary approval. Approve rejects. It’s similar to kind of what you’d see for MCP tools of approving that an action can take place. Yeah. But I think what we’ve seen with much more complex workflows from our users is that it’s actually quite advanced, like human in the loop interactions. Sometimes these could be over the course of weeks, right? It’s not just kind of simple approval for tool. There’s actual decision making involved in it. And I think as we work with those customers, we definitely want to continue to go deeper onto those use cases, too.

Alessio [00:12:21]: Yeah. What’s the entry point? So are developers also supposed to come here and then do the two-code export? Yeah. Or just segment the use cases? Yeah.

Christina [00:12:31]: So I think the two reasons that you would come to Agent Builder are one, kind of more as a playground, right? To kind of model and iterate on your systems and write your prompts and optimize them and test them out. And then you can export it and run it in your own systems using Agents SDK, using kind of other models as well. The second would be kind of to get all of the benefits of us deploying that for you, too. So you can kind of use maybe like natural language to describe what type of agent you want to build. Model it out. Bring in subject matter experts so that you really have this canvas for iterating on it and getting feedback. You know, building data sets and kind of getting feedback from those subject matter experts as well. And then being able to deploy it all without needing to handle that on your own. And that’s a lot of the philosophy around how we’re building it with ChatKit as well, right? You can kind of take pieces of it. You can have a more advanced integration where it’s much more customized. But you also get a really natural path of going live. With really kind of easy defaults as well. Yeah.

Alessio [00:13:32]: Do you see it as a two-way thing? So I build here. I go to code. Then maybe I make changes in code. And then I bring those changes back to the agent builder.

Christina [00:13:41]: I think eventually that’s definitely what we want to do. So maybe you could start off in code. You could bring it in. We’ll also probably have like ability to, you know, run code in the agent builder as well. And so I think just a lot of flexibility around that.

Sherwin [00:13:54]: The one thing I’d say, too, is a lot of the demos. The demos that we showed today I think were like, you know, aired on the side of Simplicity just so that the audience could kind of see it. But like if you talk to a lot of these customers, like they’re building like pretty complex. Like you got to like zoom out on that canvas quite a bit to kind of like see the full flow. And then for us, you know, we were kind of like working with a lot of customers who are doing this. And then, you know, if you turn that into like an actual agent SDK like file, it’s like pretty long. And so we saw a lot of like benefit from having the visual set up here, especially as the set up grows longer and longer. It would have been a little difficult to kind of showcase this. Yeah. You can do it in eight minutes. Right. Yeah. You can do it in eight minutes. But like even with some of the presets that we have on the site. Yeah, exactly. So one of the things. Yeah. Yeah.

Christina [00:14:33]: One of the things that we launched today as well alongside just like the canvas is a set of templates that we’ve actually gathered from our engineers who are working in the field with customers directly of like the kind of common patterns that they have in our own basically like playbooks when we’re working with customers on customer support, document discovery. And so kind of publishing those as well.

swyx [00:14:54]: Data enrichment. Yeah. Yeah. Planning helper, customer service, structured data Q&A, document comparison. That’s nice. Internal knowledge assistant. Yeah. Yeah. Yeah.

Christina [00:15:01]: And I think like we just plan to add more to those as we can kind of build those out.

swyx [00:15:06]: I always wonder if there should be, so you’re not the only agent builders, but obviously by default of being an open AI, you are a very significant one. Any interest in like a protocol or like interop between different open source implementations of this kind of pattern of agent builder?

Sherwin [00:15:23]: I think we’ve thought about it. Especially around, I’d say agents SDK. I would actually say maybe even like zooming out a bit more from just this is like, yeah, we were also sitting here and kind of like observing like things being made over and over again. Even like besides like agent workflows, we’re kind of launching what the industry is trying to do with responses, like what we’ve done with responses API, like stateful APIs. And so, you know, obviously we were the first one to launch responses API, but like a couple of other people have kind of adopted, I think Grok has it in their API. I think I saw LMSYS just did something recently in walls, but not, you know, not everyone. And so, unfortunately I don’t have a great answer today of like yes or no, but we are kind of like assessing everything and trying to see like, hey, you know, there has been a lot of value with MCP, with hopefully with our commerce protocol as well. ACP, yeah, that’s the, I definitely did not forget the name.

Sherwin [00:16:20]: And so like even thinking about like what we want to do with agents. Yeah. So I think that’s kind of like the same thing with the agent workflow, the portability story around that, as well as the portability, I’d say even of like responses API would be great if, you know, that could be a standard or something. And developers don’t need to, you know, like build three different stateful API integrations if they want to use different models. Yeah.

Christina [00:16:38]: And I think that’s one of the, so it’s not exactly a protocol, but one of the things that we launched today with evals too is ability to use like third party models as well and kind of bring that into one place. And so I think definitely kind of see where the ecosystem is at, which is, you know, using multi-models and kind of having.

swyx [00:16:55]: Third party models as in non-open air models? Yeah.

Sherwin [00:16:57]: Yeah. It’ll work with evals starting today. Yeah. Okay. Got it. We have a really cool setup with open router where we’re working with them and then you can bring your open router setup. And then with that, you can actually, you know, you write your evals using our data sets tool or user data set tool to create a bunch of evals. And you’d actually be able to hit a bunch of different model providers. Yeah. You know, take your pick from wherever, even like open source ones on together and see the results in our product.

swyx [00:17:24]: Yeah. That’s awesome. Speaking more about evals, right? Like I think I saw somewhere in the release docs that you basically had to expand the evals products a little bit to allow for agent evals. Maybe you can talk about like what you had to do there.

Sherwin [00:17:41]: Yeah.

Christina [00:17:43]: Yeah.

Sherwin [00:17:43]: I was going to say, so the, I actually think agent evals is still a work in progress. Yeah. I think we’ve like made maybe 10% of the progress that we need here. For example, I think we could still do a lot more around multimodal evals. But the main progress that we made this time was kind of allowing you to take traces. So the agent SDK has like really nice traces feature where if you run, if you define things, you can have like a really long trace. Allowing you to use that in the evals product and be able to grade it in some way, shape or form over the entirety of what it’s supposed to be doing. I think this is step one. Like I think it’s good to be able to do this. But I think our roadmap from here on out is to, you know, really allow you to break down the different parts of the trace and allow you to eval and like kind of like measure each of those and optimize each of those as well. A lot of times this will involve human in the loop as well, which is why we have the human in the loop component here too. But if you kind of look at our evals product over the last year, it’s been very simple. It’s been much more geared towards this like simple prompt completion setup. But obviously, as we see people doing these longer agentic traces, like, you know, how do you even evaluate a 20 minute task correctly? And it’s like, it’s a really hard problem. We’re trying to set up our evals product and move in that way to help you not only evaluate the overall trajectory, but also individual parts of it. Yeah.

swyx [00:19:01]: I mean, the magic keyword is rubrics, right? Everyone wants LM as judge rubrics. Yeah. Yeah.

Sherwin [00:19:07]: Yeah.

swyx [00:19:08]: Obviously where this will go. Okay, great. Yeah. The other thing I think online, I see the developer community very excited about is sort of automated browser. So, you know, we have automated prompt optimization, which is kind of evals in the loop with prompts. What’s the thinking there? Where’s things going?

Christina [00:19:22]: Yeah. So, we have automated prompt optimization, but again, like, I think this is an area that we definitely want to invest more in. We, I think, did a pretty big launch of this when we launched GPT-5 actually, because we saw that it was pretty difficult as new models come out to kind of learn all the quirks about a new model. Yeah, the prompt optimization. Right. There’s like, we have a big prompting guide, right, for every model that we launch. And I think building out a system to make that a lot easier. We definitely want to tie that in like completely with evals. We should be able to kind of improve your prompts over time, improve your agents over time as well if they’re kind of made in the agent builder based on the evals that you’ve set up. And so, I think we see this as like a pretty core part of the platform of basically suggested improvements to the things that you’re building.

Sherwin [00:20:05]: I actually think it’s a really cool time right now in prompt optimization. I’m sure you guys are seeing this too. It’s like not only there are a lot of products kind of like gearing around this, so like kind of what we’re thinking about. But I also think like there’s a lot of interesting research around this. Like GEPA with like the Databricks folks are actually doing really cool stuff around this. We’re obviously not doing any of the cool GEPA optimization right now in our product. But we’d love to do that soon. And also, it’s just an active research area. So like, you know, whatever Matei and the Databricks folks like might think about next, what we might, you know, think about internally as well. Whatever new prompt optimization techniques come out, I think we’d love to be able to have that in our product as well. And yeah, and it’s interesting because it’s coming at a time when people are realizing that prompt. You know, like I feel like two years ago people were like, oh, at some point prompt, like prompting is going to be dead. No. Like, you know, and it’s like, you know. It’s gone up. Yeah, yeah, yeah. And if anything, it is like become more and more entrenched. And I think that, you know, there’s this interesting trend where like it’s becoming more and more important. And then there’s also interesting cool work being done to like further entrench like prompt optimization. And so that’s why I just think it’s like a very fascinating, you know, area to follow right now. And also was an area where I think a lot of us were wrong two years ago. Because if anything, it’s only gotten better. It’s only gotten more important.

swyx [00:21:16]: Yeah. I would say like what Shunyu used to work at OpenAI and now is an MSL. We call this kind of like zero gradient fine tuning or zero gradient updating because you’re just tweaking the prompts. But like it is so much prompt that is actually like you end up with a different model at the end of it.

Sherwin [00:21:33]: There’s a lot of like things that make it more practical to just like even from our perspective, like we have a fine tuning API. And like it is extremely difficult for us to run, you know, and serve like all of these different snapshots. Like, you know, Laura’s great MSL just, you know, sorry, Thinking Labs just published. John Schulman just had a cool blog post about this. But like, man, it is like pretty difficult for us to like manage all of these different snapshots. And so if there is a way to like hill climb and yeah, do this like zero gradient like optimization via prompts, like, yeah, I’m all for it. And I think developers should be all for it because you get all these gains without having to do any of the fancy fine tuning work.

swyx [00:22:10]: Since you are part of the API team, you know, you lead the API team. And since you mentioned Thinky, I got to throw a cheeky one in there. What do you think about the Tinker API?

Sherwin [00:22:21]: yeah, it’s a good one. So it’s actually funny. When it launched, I actually DMed John Schulman. And I was like, wow, we finally launched it.

swyx [00:22:28]: Because you used to work with him.

Sherwin [00:22:29]: Yeah. Yeah. So we, it’s actually funny. So at, yeah, so right when I joined OpenAI, like this has actually been, I think, a passion project of John’s. Like he’s been talking about doing something in this. Like in this shape for a while, which is like a truly like low level research, like fine tuning library. And so we actually talked about it quite a bit when he was at OpenAI as well. It’s actually funny. I talked to one of my friends who said that when he was at Anthropic, he also, you know, worked on this idea for a bit.

swyx [00:23:01]: He’s a man on a mission.

Sherwin [00:23:03]: Yeah. I mean, John’s like so great in this regard. He’s like so purely just like interested in the impact of this because it’s, one, it’s like a really cool problem. And then two, it also empowers builders. And researchers, like you saw all the researchers who like express all this love for Tinker because it is a really great, great product. And so I’m just really happy to see that they shipped it. And I think he was really happy to kind of get it out there in the world as well. Yeah.

swyx [00:23:26]: This is probably, this is very much a digression. But like it’s weird, as someone passionate about API design, that it took this long to find a good fine tuning API abstraction, which is effectively all he wanted. He was like, guys, like I don’t want to worry about all the infra. Like I’m a researcher. I just want these four functions. And I think it’s kind of interesting. Yeah.

Sherwin [00:23:44]: Yeah.

Alessio [00:23:44]: Cool. Before the OpenAI comms team barges in the room. I know. So what feedback do you want from people like the agent builder? For example, the thing I was surprised by was the if-else blocks not being natural language and using the common expression language. I’m sure that’s something already on your roadmap. What are other things where you’re kind of like at a fork that you would love more input on?

Christina [00:24:06]: I think like one of the things that we spent a lot of time discussing was this. I think like one of the things that we spent a lot of time discussing was like whether we want kind of more of like the deterministic workflows or more LLM driven workflows. And so I think like getting feedback on that. Honestly, having people model existing workflow. A lot of what we did was kind of work with our team on, especially with engineers who are working with customers, like modeling the workflows that already exist in the agent builder and like what gaps exist, like what types of nodes are really common and how can we like add those in. I think that was that would be like the most helpful feedback to get back. And then like, you know, I think that’s kind of what we did. And then as we expand kind of from just like chat based, like right now, the initial deployment for agent builders through chat kit, we plan on kind of releasing more standalone like workflow runs as well. And kind of the types of like tasks that people would like to use in that type of API.

swyx [00:24:59]: So like more modalities, for example.

Christina [00:25:02]: Yeah, I mean, I think like for sure, like more modalities like, you know, I think kind of voice would be is already something that a lot of people have talked to us about. Even today at Dev Day. So I think modalities for sure, but also more like the logical nodes of what can’t be expressed today. Yeah.

swyx [00:25:20]: Well, you know, you’re building a language, right? You have common expression language, which I never heard of prior to this. I thought this was this Python, this is JavaScript. And then there was like a whole link in there. Was that a big decision for you guys? You know?

Christina [00:25:34]: I think that was more just kind of like a way that we thought we could kind of represent a mix of like the variables and I don’t know, like conditional statements. Yeah.

swyx [00:25:41]: The other thing I’ll also mention is that you let once you so there’s a trope in developer tooling where like anything that can be that can store state will eventually be used as a database, including DNS. So to be prepared for your state store to become a database. I don’t know if there’s like any limits on that because people will be using it.

Sherwin [00:26:01]: It’s actually funny. I’d heard this quote before and there’s definitely some truth to it. I don’t know if our stateful APIs have become a database just quite yet. But like, who knows? Like, you know.

swyx [00:26:12]: I mean, conversation. Well, you charge for it. You charge for assistance. Storage.

Sherwin [00:26:16]: Yeah. The storage. Yeah. Yeah. Right. So there’s some limit on that. But like. Yeah. But it’s very cheap. It’s like I remember we priced it like.

Christina [00:26:21]: I think if you wanted to kind of like dump all your data somewhere. I don’t know. This is like the most like transforming it all into this shape. It’s like useful. It’s easy. Best place for it. But yeah.

Sherwin [00:26:30]: But also, please don’t do this because I think it’ll put quite a bit of strain on on Benton and our input team and what we try and do. So, yeah.

Alessio [00:26:37]: How do you think about the MCP side? So you have open AI first party connectors. You have third party preferred, I guess, servers, you will call them. And then you have open ended ones. Do you see the that part of registry like functionality expanding or do you see most of it being user driven? Auth is like the biggest thing. Like if you add Gmail and Calendar and Drive, you have to like auth each of them separately. There’s not like a canonical auth. What’s the thinking there?

Christina [00:27:03]: Yeah. I mean, I think definitely for the registry. That’s why we want to make it a lot easier for like companies to kind of manage what they’re like. Developers have access to managing kind of the configurations around it. And I think in terms of like first party versus third party, like we want to support both of those. We have some direct integrations and then anyone can kind of create MCP servers. I think we want to make that a lot easier to like establish kind of private links for for companies to use those internally. So I think like just really excited about that ecosystem growing. Yeah.

Sherwin [00:27:35]: I think one of the coolest things observed, too, is just I actually think we as an industry are still trying to figure out the ideal shape of connectors. So, I mean, part of why I think the 1P connectors exist, too, like we end up storing quite a bit of state. It’s like a lot of work for us. But like by having a lot of state on our side, we call them sync connectors. We can actually end up doing a lot more creative stuff on our side when you’re chatting with ChatGPT and using these connectors to kind of boost the quality of how you’re using it. Right. Like if you have all the data there, you can do all this like re-ranking. You can like we can put in a vector store if you want to put it anywhere else. Whereas and so there’s some inherent tradeoffs here where like you put in a lot of work to get these like 1P connectors working. But because you have the data, you can do a lot more and get higher quality. But then but then the question is like, oh, my God, there’s like such a long tail of other things, which is where the MCP and like the third party connectors come in. But then you have the tradeoff of like you’re beholden to like the API shape of the MCP creator. It might actually work well, might not work well with with the models. And then what happens if it doesn’t work well? Then you kind of have to like, you know, you’re kind of like at the mercy. Of this and MCP. But by the way, it’s like really great because it already does some layer of standardization. But my senses are still going to be more evolving here. And I think, you know, we want to support both of them because we see value in both right now, especially working with working with developers. You want to have kind of like all options kind of on the table here. But it will be interesting to see how see how this evolves over time.

Alessio [00:28:54]: Yeah. When I saw about three, four months ago when you launched the forum for like signing with chat GPT interest. I think to me, that’s kind of like the vision where I log in. And I have the MCPs tied in and then I sign in with chat GPT somewhere and I can run these workflows in that app where I’m logging in. So, yeah, I think Sam, you know, said in an interview that is chat GPT is like your personal assistant. So I think this is like a great step in that direction. Yeah. I think there’s a lot more to go in that in that direction.

swyx [00:29:22]: But so far, no plan on like chat GPT or opening as IDP. Right. Which is a different role in the in the off ecosystem. Yeah.

Sherwin [00:29:31]: It’s interesting because. So direct answer is like no plans right now, of course. But I actually think we currently have some version of this, which is our partnership with Apple. Because with Apple, you can actually sign in to your chat GPT account. And some of that identity does carry with you into your iOS experience with Siri. Right. Like if you if you I don’t know if you’ve actually used this, the Siri integration. I actually use it quite a bit. But if you sign into your chat GPT account, the Siri integration will actually use your subscription status to decide. What type of model to use when it when it passes things over to chat GPT. And so if you’re, you know, just a free user, you get, you know, the free model. But if you’re a plus or pro subscriber, you get routed to GPT-5, which is I think what they.

Christina [00:30:17]: I think we also recently announced the partnership with Kakao.

Sherwin [00:30:19]: Oh, yeah. Kakao is another one. Yeah.

Christina [00:30:20]: Where I think it’s a similar thing where you can sign in with chat GPT. Kakao is one of the largest like messenger apps in Korea and kind of interact with Kakao directly there. Yeah.

Sherwin [00:30:31]: I mean, Sam’s been talking about it for a while. It’s a very compelling vision. We obviously want to be very thoughtful with how we do it.

swyx [00:30:36]: I mean, you know, now you have a social network, you have a developer platform. My you know, my chat GPT account is very, very valuable. Yeah. Yeah, exactly. OK. So and then on the other side of the office, something I was really interested to look at and I couldn’t get a straight answer. Is there some form of bring your own key for AgentKit? Like when I when I expose it to the wider world, obviously, like I mean, by default, I’m paying for all the inference. But it’d be nice for that to have a limit. And then if you want more, you can bring your own key.

Christina [00:31:07]: Yeah. I mean, we don’t have something like that yet. But I think, yeah, it’s definitely an interesting area, too. Yeah.

Sherwin [00:31:14]: It doesn’t do it out of the box today. But, you know, developers have been asking about it for forever. Like it’s a really cool concept because then as a developer, you especially any developer, you don’t need to bear the burden of inference. Yeah.

swyx [00:31:26]: I think like when you get into the business of like agent builders that are publicly exposed, where you have like an allowance. Yeah. You know, there’s a list of domains like this is this is the it rhymes with this exact pattern of like someone has to bear the cost. And sometimes you want to mess around with like the different levels of responsibility. Yeah.

Sherwin [00:31:43]: I will say in general, like if you kind of look at our roadmap, we engage a lot with developers. We kind of hear what are the pain points and we try and build things that address it. And, you know, ideally, we’re prioritizing in a way that’s that’s helpful. But, yeah, we’ve definitely heard from a good number of developers that like the cost is we’re like all of the like copy paste your key solutions. Right now, which are like huge security issues like hazards because developers don’t want to bear the burden of of inference. You know, hopefully we make the cost cheaper.

Christina [00:32:08]: So it’s the models keep getting cheaper. Yeah. Hopefully that helps.

Sherwin [00:32:12]: But but what we realize is as we make it cheaper, you know, the demand for that goes up even more and you end up still spending quite a bit. But yeah, so we definitely heard this from a lot of developers and it’s definitely something top of mind. Yeah.

Alessio [00:32:23]: Do you see this as mostly like an internal tools platform, though? Like to me, like you’ve been doing a big push on like the more forward deployed engineering thing. So it’s almost like, hey, we needed to build this for ourselves as we sell into these enterprises might as well open it up to everybody. What drives drive building these tools? Like you think of people building tools to then expose or mostly on the internal side? Yeah.

Christina [00:32:45]: I mean, it’s so like I think, again, our first deployment is ChatKit, which is kind of one of it’s intended to be for external users. But I think one of the things that we also did see a lot as we were working with customers is that a lot of companies have actually built some version of an agent builder internally. Right. To kind of manage prompts internally, to manage templates that they’re sharing across, you know, the different developers that they have, maybe the different product areas. And we were seeing that kind of like over and over again as well and really wanted to build a platform so that this is not, you know, an area that every company needs to invest in and like rebuild from scratch. But that they can kind of have a place where they can manage these templates, manage these prompts and really focus on the parts of agent building that is more unique to their business.

Sherwin [00:33:29]: It is interesting, too. Like from a deployment perspective, it is like it has spanned both internal and external use cases, right? Like kind of like these internal platforms, people will use it for like data processing or something, which is an internal use case. But if you saw some of the demos today, like there have been a huge number of companies that are trying to do this for external facing use cases as well. Customer service is one template in here. Customer service, the like ramp use case.

Christina [00:33:52]: We use this internally and externally. Like our customer support, help.openair.com is already powered on AgentKit and then various other like internal use cases.

Sherwin [00:34:00]: And one of the things that I actually think the team has done a really great job of, so like Tyler, David and Jiwon on the team, they built the, especially the ChatKit components, they built it to be like very consumer grade and like very polished. Like you kind of look at that, there is like a whole grid of like the different widgets and things that you could create there. Like ideally people see it and they see it as like these very polished like consumer grade ready external facing things versus like, you know, you think of internal tools and like the UI is always like the last thing that people care about. Yeah. So like you really, you know, push the team. And I think they did a really great job of making the ChatKit experience like really, really consumer grade. And it should feel almost like ChatGPT and with like really buttery smooth animations and like really responsive designs and all of that. Yeah.

Christina [00:34:43]: I think your point on widgets is like definitely like really resonates, right? Because ChatKit, it handles the chat UX, but we’re also just building like really visual ways for you to represent like every action that you want to take. And that is definitely like very high polished.

Sherwin [00:34:59]: Yeah. And when working with customers, like those have been the most helpful customers for us to work with because, you know, when Ramp is thinking about, you know, how, what they want to publicly present to people, like they have a pretty high bar as they should, as well as, you know, all the other customers that have been iterating on it. And so that kind of feedback from our customers has really helped us up level the general product quality of the launch that we had today as well.

swyx [00:35:19]: Yeah. Would you ever, would you open source ChatKit?

Sherwin [00:35:25]: Talked about it. We’ve talked about it. There are a bunch of trade offs. Yeah.

Christina [00:35:30]: I think so. So ChatKit itself is like an embeddable iframe. And so I think the actual. Oh, it’s an iframe. Yeah. And so that helps us keep it like evergreen. Right. So if you are using ChatKit and we come up with new, I don’t know, a new model that reasons in a different way, right, or a kind of new modalities that you don’t actually need to rebuild and like pull in new components to use it in the front end. I think there’s parts of, you know, widgets, for example, that is much more like a language and can definitely is something that is easier to explore that for, as well as kind of the design system that we’ve built for ChatKit. But I think like as part of, yeah, the actual iframe itself, I think there’s a lot of value in that being, yeah, more evergreen. Yeah. More evergreen experience that is pretty opinionated.

swyx [00:36:13]: Like there’d be no point in being open source. Right. Yeah. You want the.

Christina [00:36:16]: Then you don’t get the benefits of it.

swyx [00:36:18]: You know, being Stripe alums, like Stripe Checkout, like it’s, it’s all optimized for you to like.

Christina [00:36:23]: So I’m not a Stripe alum, but Kristina is. Right. And the team actually is the team that built.

Sherwin [00:36:28]: Stripe Checkout.

Christina [00:36:29]: Yeah. So it’s very similar philosophically. Right. So Stripe, you know, can build elements and checkout and not every business needs to rebuild, right, the pieces that are really common. And I think we see the same with chat. We see chat being built over and over again. Especially when you’re building a website. Especially as we kind of come up with new, you know, modalities, like reasoning, everything. It’s not really something that is easy to keep up to date. And so we should just do that and leave kind of the hard parts of building agents again to.

Alessio [00:37:02]: Yeah. To the developers. Does it feel, I mean, I know WordPress is like a bad connotation in a lot of circles, but to me it almost feels like the WordPress equivalent of like chat is like, hey, this is like drop in thing. And then you have all these different widgets. Do you see the widget becoming a big kind of like developer ecosystem where people share a widget? Is that kind of like a first party thing? And then what’s like the M4 versus widget forest? No, exactly. I mean, it’s kind of like it seems great for people that are like in between being technical and like not really being technical enough. Yeah. Yeah.

Christina [00:37:35]: Yeah. I mean, I think that’s a big part of building widgets, right? Like it’s already kind of in the language that is very consumer friendly. You can use in our widget builder already. You can kind of use AI to create those widgets and they look pretty good. I don’t know if you guys have gotten a chance to try that out yet, but definitely see kind of, I don’t know, a forest.

Sherwin [00:37:53]: If you haven’t tried out the widget studio and the demo like apps as well. Yeah.

swyx [00:37:59]: You got a custom domain like widget.studio. It’s cool.

Sherwin [00:38:02]: I actually don’t know how we got that. Yeah.

Christina [00:38:04]: Everything’s in chatkit.studio. And then we have like the playground there so you can try out what chatkit would look like with all the customizations. We have chatkit.world, which is a fun site we built. I was like spinning the globe for a while this morning. It was like a fidget spinner.

swyx [00:38:18]: I think Kasia also like uploaded some of her solar system stuff and all the demos as well. Yeah.

Christina [00:38:24]: And then that’s where like the widget builder. Yeah.

swyx [00:38:26]: So it’s really come together. It’s taken like almost more than a year to like come together and like build all this stuff, but it’s coming together. It’s like really interesting.

Sherwin [00:38:34]: Yeah. Yeah. It’s something that we like- You definitely planned all of this upfront. Oh yeah. Yeah. We have the master plan from three years ago. No. But like I think, especially on this stuff, I think there was like an arc of a general like, you know, platform that we did want to kind of build around. And it takes a while to build these things. Obviously, Codex helps speed it up quite a bit now. But yeah, I will say it does seem great to kind of like start to have all the pieces start fitting together. Yeah. I mean, you saw we launched evals and we had the fine tuning API for a while and we laid all the groundwork for some of the stuff over the last year. And we’re hoping that we can eventually, you know, make it into this full feature platform that’s helpful for people.

swyx [00:39:11]: I think you have. Since you did the Codex mention, maybe a quick tip from each of you on Codex power user tools or tips.

Sherwin [00:39:23]: So there’s actually a funny one that one of the new grads has, I think, like taught our team in general. And I think this is like a point for like just how like new grads and younger, you know, generation people are actually more AI native. So one of them is to like really lean into like- Yeah. Like push yourself to like trust the model to do more and more. So like I feel like the way that I was using Codex. And so for me, it’s usually for my personal projects. They don’t let me touch the code anymore. But you give it like small tasks. So you’re like not really trusting it. Like I view it as like this like intern that like I really don’t trust. But what a lot of the like, so we had an intern class this year. What a lot of the interns would do is just like full YOLO mode, like trust it to like write the whole feature. And it like it doesn’t work. It doesn’t work for worse. It like doesn’t work sometimes. But like, I don’t know, like 30, 40% of the time, it’s just like one-shots it. I actually haven’t tried this with like Codex, GBD-5 Codex. I bet it probably like one-shots it even more. But one tip that I’m like starting to like, I feel like undo this, like relearn things here is to like really lean into like the AGI component of it and just like really let the model rip and like kind of trust it. Because a lot of times they can actually do stuff that surprises me. And then I have to like readjust my priors. Whereas before I feel like I was in this like safe space of like I’m just treating this, I’m giving this thing like a tiny bit of rope. Yeah. And because of that, I was kind of limiting myself with how effective I could be.

swyx [00:40:44]: Like, sure, but okay. But also, is there an etiquette around submitting effectively, you know, vibe-coded PRs that someone else now has to review, right? And it’s like, it can be offensive.

Sherwin [00:40:55]: Well, we have Codex who reviews now.

swyx [00:40:56]: Okay. It actually reviews itself. Does Codex approve its own PRs a lot more than humans? It doesn’t get approved then.

Christina [00:41:02]: I was going to say, I think like the Codex PR reviews are actually one of like the things that my team like very much relies on. I think they’re very, very high quality reviews. Yeah. On the Codex PR side, like for the visual agents builder, we only started that probably less than two months ago. And that wouldn’t be possible without Codex. So I think there’s definitely a lot of use of Codex internally and it keeps getting better and better. And so, yeah, I think people are just finding they can rely on it more and more. And it’s not, you know, totally vibe-coded. It’s still, you know, checked and edited, but definitely has a kicking off point. And I think I’ve heard of people on my team, it’s like on their way to work, they’re like kicking off like five Codex tasks because the bus takes 30 minutes, right? And you get to the office and it kind of helps you orient yourself for the day. You’re like, okay, now I know the files. I have the rough sense. Like maybe I don’t even take that PR and I actually just like still code it. But it helps you just context switch so much faster too and be able to like orient yourself in a code base.

Sherwin [00:42:01]: There are so many meetings nowadays where I have like one-on-ones with engineers and I walk into the room, they’re like, wait, wait, wait, give me a second. I got to kick off my like Codex thing. I’m like, oh, sorry. We’re about to enter async zone.

Christina [00:42:10]: It’s almost like your notes, right? You’re like, let me.

Sherwin [00:42:12]: And they’re like typing like, okay, now we can start our one-on-one because now it’s great. Yeah.

swyx [00:42:16]: Cool. We’re almost out of time. I wanted to leave a little bit of time for you to shout out the Service Health Dashboard because I know you’re passionate about it. Well, tell people what it is and why it matters. Yeah.

Sherwin [00:42:27]: So this is a launch that we actually didn’t, you know, it didn’t get any stage time today, but it’s actually something I’m really excited about. So we launched this thing called the Service Health Dashboard. You can now go into your usage or like your settings account and kind of see the health of your integration with our OpenAI API. And so this is scoped to your own org. So basically if you have an integration that’s running with us doing a bunch of, you know, tokens per minute or a bunch of queries, it’s now tracking each of those responses, looking at your token velocity, TPM that you’re getting, the throughput, as well as the responses, the response codes. And so you can see kind of like a real-time personal SLO for your integration. The reason why I care a lot about this is obviously over the last year, we’ve spent a lot of time thinking about reliability. We had that really bad outage last December, you know, longest, like three, four hours of my life, and then had to, you know, talk to a bunch of customers. We haven’t had one that bad since, you know, knock on wood. We’ve done a bunch of work. We have an Infer team led by Venkat, and they’ve been working with Jana on our team, and they’ve just been doing so much good work to get reliability better. And so we actually... Yeah. Again, knock on wood. We think we’ve got reliability in a spot where we’re, like, comfortable kind of putting this out there and kind of, like, letting people actually see their SLO. And hopefully, you know, it’s, you know, three, four, soon to be five nines. But the reason why I cared a lot about it is because we spent so much time on it, and we feel confident enough to kind of have it behind a product now. Five nines is like two minutes of outage or something. Yeah, yeah. We’re working to get to five nines. What? What does an extra nine take?

Sherwin [00:44:08]: It’s exponentially more work. So, you know, and then... But, like, we always... We were, you know, in the last couple weeks, we were talking about, like, hitting three nines and then hitting three and a half nines and then hitting four nines. But, yeah, it’s exponentially more work. I could go for a while on the different topics, but...

swyx [00:44:25]: We’ll have to do that in a follow-up. I mean, that’s all... That’s the engineering side, right? Yes, yes, yes. Like, you’re serving six billion tokens per minute.

Sherwin [00:44:32]: We actually zoomed past that. Yeah, that’s the... That’s outdated. Yeah. But, yeah, it’s been crazy, though, the growth that we’ve seen.

Alessio [00:44:39]: Awesome. I know we’re out of time. It’s been a long day for both of you, so we’ll let you go, but thank you both for joining us.

There’s nothing as exciting and scary as a blank page. Anything is possible, but where to begin? In the age of AI, the kickstarting process is more and more being delegated to a prompt. “Design an app to do X” or “Write me a blog post on Y”. This is the root of all slop.

In our Greg Brockman episode we talked about the tendency that LLMs have to create purple and blue gradients due to their training data. Today’s guest is the man leading the company that might help us fix this: @zoink aka Dylan Field, CEO of Figma.

We have often talked about the software engineering triad of specs, test, and code. After my discussion with Dylan, I have adapted this to the design world through levels of composability instead: the full design, the individual components, and the underlying design system. Each of the three has a corresponding product slice:

• Figma Make: This is the highest level of abstraction; you can go from prompt to full design and working app just like Lovable, Bolt, etc. All changes are made via prompts in chat, but the design can be moved down one level of abstraction to a Design.

• Figma Design: The core design product. Any Make product can be turned into a Design, and any Design can be turned into a product through Make. This allows any designer to make pixel-level changes to each component that can then be propagated to the actual code.

• Design system: the rise of Tailwind CSS and named classes makes design systems extremely easy to use now for most applications. Figma has a similar concept with variables; using something like Tokens Studio you can use a json in your Github repo to sync your code and your Figma variables for consistent style. Similarly, all Design changes can propagate down to a design system and then into your code. (We also talked about CodeConnect for larger codebases)

“Figma to Code” was its own category of early stage startups for a bit. Once vibe coding started to rise, there was also a lot of chatter about design being the new bottleneck. Figma is now the Figma to Code company, and they are leaning into that.

I have three MCP servers always on: the Figma server, Sentry, and Linear. Once you have done the work above, you can just ask your coding agent to implement the design changes.

We also had a great discussion about design and the future of software, what he learned from the Thiel Fellowship, and how he first got AI-pilled. Enjoy!

Subscribe now

• Figma

• Figma Make

• Dylan Field

• Pete Skomoroch

• Evan Wallace

• John Doerr

• Warp

• Chris Olah

• Chain Runners

• [00:00:00] Figma’s Mission: Bridging Imagination and Reality

• [00:00:56] Becoming AI-Pilled

• [00:07:44] Figma Make

• [00:08:57] Language as the Interface for Design

• [00:13:37] Source of truth between design and code

• [00:18:15] Figma as a Context Repository

• [00:21:30] Understanding and Representing Design Diffs through AI

• [00:24:20] Figma’s Role in Shaping Visual Aesthetics

• [00:31:56] Fast Fashion in Software

• [00:36:04] Limitations of Prompt-Based Software Creation

• [00:39:43] Interfaces Beyond Chat

• [00:42:12] Lessons from the Thiel Fellowship

• [00:44:58] Using X for Product Feedback

• [00:48:10] Early-Stage Recruiting at Figma

• [00:53:11] Positioning Figma Make in the Prompt-to-App Landscape

• [00:55:19] Digital Scarcity & AI

Alessio [00:00:00]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, founder of Kernel Labs, and so happy to be at the Figma office today with Dylan Field. Welcome.

Dylan: Thank you. Thanks for having me on the podcast and welcome to the Figma office.

Alessio: Yeah, you know, we almost couldn’t choose where to do this because there’s so many beautiful spaces in it, but we finally had a way with this corner. Super excited to have you on today. I was reading through some of the history of Figma and your initial mission was, you know, to close the gap between imagination and reality. And if I heard that today, I would assume it would be the slogan of one of the vibe coding platforms. And so maybe talk about what was like the first, we should take AI seriously moment where you were like, okay, imagination to reality in the first phase of Figma was like helping designers bring what they had in their mind into a canvas. And now with Figma Make, you’re obviously moving to like a much broader audience. So what was the journey to get there?

Dylan [00:00:56]: Yeah, I mean, I think if you go back far enough, you know, AI showed up in different. Forms for Figma. So I had the chance to be on the data science team at LinkedIn as an intern prior to working at Flipboard and gave more into design and the story in Figma. And we were doing, you know, a lot of more classical machine learning approaches. And I was kind of absorbing that. And there’s plenty of discussion about agents back then with my mentor, Pete Scomrock, and thinking through, okay, what might it look like if some of the ideas from the 90s. Were to resurface. And, you know, those were just kind of like fun, geeky conversations that are pretty abstract because obviously the world wasn’t there yet. And then back at Brown with Evan, my co-founder and our original CTO, who’s no longer at Figma, but an absolute legend. I mean, just check out his GitHub if you’re not convinced of that. He and I were talking a lot about some of the stuff we’re starting to see as sort of ML and computational photography approaches to doing image editing. And what could be accomplished with that? So, for example, there were papers being written about how do you use Internet scale data to complete scenes and make it so you can basically do the equivalent of like content-aware fill. But instead of doing it in an algorithmic deterministic way, how do you do that based on the entire Internet? And we thought that was like a pretty fascinating concept. And there’s a professor at Brown who was doing some cool research in this area. We also were getting very excited in the early days of Figma before we even incorporated. Like, how do you turn a 2D image into a 3D scene? So more computational photography, you know, plus on blending and some of these early techniques that you kind of get like 85% of the way there to something awesome, but not 100%. And it wasn’t until, you know, we really had deep learning that you could get to 100%. But all of these individual demos that we’re able to work on and by we, I mean, mostly Evan, he’s the real genius in the equation here. But as we started to explore... We started to explore a bunch of these areas, it just felt like there must be some way to make creation easier. And so that’s why it’s the vision was data’s idea to reality and not like idea to X as a subset of reality, because we thought actually you could do this for a lot of different areas. And I still do. But we’re starting with a data product and fast forwarding to today, Figma Make, for example, we’re really trying to make it so that you can go from idea in your head to actual reality. And so we’re starting with a data product and fast forwarding to today, Figma Make, for example, we’re really trying to make it so that you can go from idea in your head to actual reality. And that might take the direction of an internal prototype to explore different ideas. It might be an internal app that you’re using. I’ve been supporting some work on like random data munging that I was using to make for it, which is kind of fun. And rather than like write a Python script. And it’s, I think, very exciting to think about how far you can help people go and how you can make them both more profitable. But also help them explore more of the options space of design with some of these techniques. And then, of course, we’re also excited about what that means in Figma design as well. How do you prompt to edit, prompt to do generation and do it in a way that’s consistent with everything else that’s in your design system, the patterns you’re already using?

Alessio [00:04:22]: And how do we actually infer from what’s already inside of Figma what you want to do and really be expansive in the way that we do it? Understand your intent. So you have a background in obviously math and CS and now you run Figma. So you have this kind of like duality of like aesthetics and code. Were you first AI-pilled by the image generation, kind of like more creative things? I think early on in the podcast, most people would say Midjourney was their favorite AI product. And another half of people would say GitHub Copilot. What was your first product that you fell in love with with AI?

Dylan [00:04:56]: Not a product, but my first like AI-pilled moment was, I think it was like 2014. 14 or so, maybe a little bit earlier. I was a Teal Fellow. And in my class was a number of amazing people, but one of which was Chris Ola. And Chris and I would be going to these retreats together for the Teal Fellowship every, you know, three to six months. And I remember one of them, Chris had been working on some cool like Haskell 3D generation stuff. And it was all a bit like out there and not clear how it would be. And I remember him sitting down with me. We were at like a wooden table outside in some like Santa Cruz, you know, nature setting. And he’s on the Wi-Fi, which is super slow, connecting to AWS. And he’s like, look at this. I can go on AWS and I can spin this up and I can train this like tiny neural net to classify. And I was like, oh, my God. I’m like, Chris, this is like a solved computer vision problem. Like, why are you excited? And he’s like, no, you don’t get it. It’s a neural net. And there’s like hyperparameters I can tweak. And I think I can actually make, you know, another neural net to like figure out how to tweak the hyperparameters. And I’m like, oh, that’s all great. But like, this is a solved problem. And I lacked the vision at that point to see where it was going. And but it started to get me to pay more attention. And then watching his work when he was at Google. Some of the great blog posts he was doing, as well as starting to listen in on more of people around me and the conversations that were happening around AI and machine learning, got me more and more excited about where this might go. But I don’t think I truly internalized scaling laws for quite a while longer and what that could mean. But I think GPT-3 was probably the first time that I was like, wow, the delts between this and past models is so great. Something. Exponential is definitely happening here. It’s not just like hype. And then, you know, plenty of conversations around that time with other AI figures that we both know well definitely started to make me think, okay, there’s something very important to focus on here. But I think it’s very different to be in a context of, you know, more deterministic software building than AI research. There are completely different motions of how you kind of run those teams. And so it definitely took us a lot longer than, you know, starting at, okay, GPT-3, amazing, to get to the point where we’re starting to ramp up and push the boundaries of what might be possible at Figma.

Alessio [00:07:44]: No, that’s great. And yeah, I would say Figma Make is one of the maybe most impressive releases I’ve seen this year. I was playing around with it the last few days. I built a Figma clone in Figma Make, so you guys are cooked. You let us realize a bit. Exactly.

Dylan [00:07:58]: Or are you so back?

Alessio [00:07:59]: I don’t know. I know. I don’t know. It’s so hard to keep up. It depends on the day. Yeah, exactly. Tomorrow that might change. But to me, there’s this interesting triad in software engineering, which is like you have the tests, you have the spec, and you have the code. Yeah. And you usually, if you have two of the three, you can generate the third. I’m curious how you think about the Figma model, so to speak, the almost Figma data model. You have Figma Design, which is like where the visual work happens. Then you have Figma Make, which is basically, in my mind, the bridge between the design and the code. And then you have the Figma MCP, which is like, how do you bring that into code in a way that it’s not even UI-driven? It’s just like the model is kind of doing the work for you. Does it feel like it’s changing, in a way, the tools that you need to build? And how do you think about, yeah, you mentioned using AI for editing the design and whatnot. Do you feel like natural language is becoming more and more the interface, even in design, that the work is going to be done? Or, yeah, what are the pieces in your mind?

Dylan [00:08:57]: Yeah, lots to unpack there. Sure. Sure. Sure. Is natural language the interface? Yes, right now. I’ve said it before, but I really believe it. I think we’ll look back on this era as like the MS-DOS era of AI. And the prompting in natural language that everyone’s doing today, I think, is just sort of like the start of how we’re going to create interfaces to explore Latent Space. So I’m just like, cannot wait for an explosion of creativity there. Because I think when I think of these models, it’s like they’re almost like an N-dimensional concept. There’s this little compass that lets you explore this wild, unknown fog of war in Latent Space. And you can kind of push the models in different directions through natural language. But if you have a more constrained end there and you’re able to dimensionality reduce a bit so you can push different ways, there should be other interfaces available than text. These might be more intuitive, but they also might be more fun to explore. And I think sometimes constraints unwind. So I’m excited for that. But right now, yes, natural language is where we’re at. And while I’m excited to push that forward, meet people where they are, I think, is usually a good model for product development before you get to the point where you’ve really refined. Going back to your triad, I think maybe we can start with the spec. Like, I think the notion of a spec is evolving so much right now. And what should be in a PRD? Versus what should be in design versus what should be in code. That is, I think, much more blurry than it used to be. It used to be that we had, obviously, this, like, very kind of waterfall-y process of, oh, yeah, we’re going to go gather some requirements and then we’re going to go, you know, make a big doc. And then we’re going to go make some designs and we’ll code it up when we feel it’s ready. Maybe you go repeat a few times, but it was a process. And I think. With Figma, you can absolutely follow that process. But also, we recognize that roles are blurring, stages are blurring. And as all that blurs, how do you actually support different ways of working? You might want to make a prototype as part of or in place of, you know, a PRD. You might actually want to focus more on the design as a high-fidelity descriptor of what this could mean if the cost to make design and to create designs is low. And I think that the more that you can kind of expand that option space for people and bring them into a surface to align design and visual fidelity might be the place where you actually can align best. And there’s also the question of, okay, how far can a spec get you? And why is a spec different than code? So if code is the complete spec in terms of it is the most determined, clear way to show up in a design, then it’s going to be the most determined way to show up in a design. So intent of what should happen in every edge case, well, how much of that can be inferred? I think that’s an open question, but one that we’ll all be thinking a lot about soon. And if you think about sort of the value stack overall, it feels to me that the better cogeneration gets, the more design matters. And the more that actually the human pushing on design matters, too. Because even if you have a good starting spot from your design system, from... You know, AI generation, whether it be code or image, you, I think, need to push design forward, not just as an individual screen, but as a system in order to actually compete, differentiate, and win. It’s been our thesis for a long time, design is a differentiator. But I think it’s even more true in this world where we’re at now, where the rate of software creation is going exponential and maybe even vertical. And in that world, you have more software, there’s more competition, so what wins? Well, it’s brand, it’s point of view, it’s taste, it’s craft, it’s design. And I think that’s, if that’s the world we’re headed for, which I’m very confident it is, then it’s not enough just to use AI to generate an output. But I think you have to push further than that and really get in the detail into the craft. And to utilize an AI to explore the option space faster, so you can go as deep as possible in the direction you choose.

Alessio [00:13:37]: Yeah, you know, I only have the pro plan, so I don’t have Code Connect. But I’m curious how you think about that. Because Code Connect, the whole idea, in my mind, was like, hey, instead of having to make sure that the code stays in sync with the design, we kind of build this bridge between the two. But now, if you have the design, you can, in theory, every time regenerate the component anyway. So why add, you know, maybe this additional layer? That before it was there, it’s not needed anymore. To me, that’s the most interesting thing. I was like, what’s going to end up being the source of truth? And what are, like, the two-way bridges? So, for example, right now in FigmaMate, I use the MCP to bring that code into Cursor, my actual code base. But there’s no way yet, I’m sure you’ll do it, for the MCP to write back into the design and say, we actually ended up implementing it this way. I’m curious where you feel like the center of gravity is going to be. Like, obviously, you’re biased in a way. But as an engineer, you know, I’m curious your thoughts. Yeah.

Dylan [00:14:30]: Well, first of all, just kind of explain CodeConnect a bit more. So, to expand on what you already said, for, I think there’s different situations that you might find yourself in. So, you might be going to zero to one, making a prototype of something that’s rather disposable. You might be actually working on a personal project where you’re not making something that’s disposable. You’re building on something that’s existing, but the code base is small. It’s, like, pretty clear what’s going on to you. And there’s not a lot of patterns that exist. Or you might be in a pretty large code base where there’s a lot of existing patterns, a lot of code. And you’re trying to fit those patterns. So, especially in that last example, I think as you get to these larger code bases and larger sort of settings inside of companies, it’s very important to be consistent with existing patterns in the code. But also, it’s important to create a design system where, you’re able to create consistency at scale for designers and make it so that people are more efficient. So, they’re not always recreating different buttons, et cetera. There’s the world of Figma and design, and there’s the world of code. And there are advantages to having a source of truth in Figma and a source of truth in code. So, in some cases, we look at libraries that customers make, and they are one-to-one. The design components perfectly reflect the code components. So, the components in the code base. In other cases, you’re working on the thing that’s next in Figma, and that’s not yet all built out in code. And both cases are important. In the case where you are one-to-one, and there’s more of that bijection between components in Figma, components in your code base, you want to define a formal mapping. So, that way, you’re able to give context via MCP, make it so that developers are easily able... to implement a design on the front-end. And CodeConnect serves that goal. We’re doing a lot of investment to make it easier to set up, because right now, it’s too much of a pain. But also trying to get further in terms of how many people can use CodeConnect. And in terms of where the source of truth lies, I think there’s a variety of ways it will probably play out in parallel. I think that it’s okay if, for example, in some times, that code is the source of truth. And you’ll see us do a lot of work to make it so that you’re able to bring your code-based design system into something like Figma Make or Figma Design. But also, if you’re wanting to rapidly iterate and be able to express and try out different visual explorations, and if you think visually, if you are someone who’s not necessarily maximal comfort with code, I think a visual surface is... very important as a place to explore. And I think there are different modes of thinking, and it might be the case, I think it’s likely the case, that the visual sort of metaphor is easier for a wider set of the population to grok than to go into code. I also think that it’s going to be something that, as we move forward in time, with more agents writing more parts of your codebase, I think that it’s going to be something that, as we move forward in time, with more agents writing more parts of your codebase, you will also be less familiar with the code. And so then you might want a different abstraction where you’re able to work on things and basically plan out what your app should be, what your software should be, and Figma can provide that.

Alessio [00:18:15]: Yeah, I almost think of Figma as like the context repository for aesthetics. To me, it’s almost as an engineer, right? The design product is not even that helpful, that useful in a way, as long as you can generate the components and I can do small tweaks. And I think one of the big tailwinds that Figma has is, pardon the pun, but tailwind CSS bringing a lot of this more classes, name classes as a way to define style, and the way the Figma variables and the way your system is set up. For me, once I saw Figma me, I’m like, okay, now I get it. Before, when I had the blank Figma canvas, it’s like, I’m not talented enough to start from here, but if you can build an initial thing through AI, then I’m good enough to tweak it and then have that be now the bridge. And then I can take that in Clock Code, I can take that in Cursor, or maybe I’ll just say Figma make forever and the prompts just go there.

Dylan [00:19:11]: Yeah, I think that what you’re saying is super important as a point. The blank canvas problem is real. We’re always trying to figure out for Figma design, how do you make it less intimidating for someone to come in? How do you make this more approachable? And there’s always tension between the power users of Figma, Figma design, who would like every single power feature that you can imagine. Why can’t you make a feature for everything in the CSS spec? And of course, we can over time. On the other side of it is, okay, you’ve got someone coming in for the first time, technical, non-technical, whatever. Are they intimidated? Do they feel invited to go create something? And the first, regardless of our UI, the first thing that can block people is that blank canvas. So getting people from the place of, you know, I have an intention to actually putting something on that canvas is so important. And I think once you start getting people in that loop, then it’s less intimidating. You have more that you want to explore. One of the things I’m excited about that we just actually shipped today is a way to copy designs from make into Figma design. And if you think about Figma make, as a just easier entry point for Figma design, you know, it’s like the flight simulator to the airplane cockpit or something, then perhaps you’re able to make it so that that’s an early entry point you go through. And then you actually can do more than just tweaking components, but actually visually manipulate a design. And you might find that you actually can go faster that way than if you’re doing it in code.

Alessio [00:20:51]: Yeah, I think, I don’t know what your model internally is between of diffs, but to me it’s like it’s easy to communicate what a diff in code is, but it’s kind of hard to communicate in design in a way that I can then put into an LLM to like apply. So I’m curious how you think about that. It’s like there’s obviously the design is more than the sum of the components, right? It’s like if you just like any piece that is in Figma individually doesn’t look like anything, then when you put them together, it looks great. How do you see the way people communicate design? Also change now that more of it needs to become language because of the interfaces.

Dylan [00:21:30]: Yeah, I think that I might push back a bit on the more of it becomes language part, but maybe we can explore that later. Depends on exactly what you mean by that. But in terms of the way that we represent diffs, you can go to version history in Figma and see sort of composites of diffs. You can make a new version at any point. And of course, internally, there is the journal of every single edit. And I think that’s really cool. I think there is opportunity there, to your point, around how do you basically use not just what’s in Figma design as a source of truth, but also the tool calls that made via MCP and when they are made to understand what is the context that’s changed since I last made a call. So that’s an opportunity that I think is a smart one to point out. But also, I think it’s just interesting to think about the journal data and what could be possible in terms of thinking about what you might do next, how to help you be more efficient and help you explore more ideas.

Alessio [00:22:32]: Yeah, my natural language part is if I give an LLM a get a progress, it can understand from the code what type of changes I’ve made. I’m trying to figure out, and this is just because I’m not a Figma power user, if I gave an LLM a Figma journal or diff, could it understand aesthetically what has changed to then update maybe other docs or parts of my system?

Dylan [00:22:58]: Yeah, right now that’s not there. I think it’s a really good idea. And I think it also mirrors other ideas that are going to be important too. For example, if you pull in Figma context into your IDE or your agent, whatever format it takes, let’s say that the design is not perfectly implemented, because currently that’s where we’re at, is a lot of times it’s, wow, this is a great start, but there’s more work to do. Well, what’s the delta? How do I get to the point where it’s perfectly implemented? And that requires some back and forth too. Yeah.

Alessio [00:23:32]: When I did the Figma to Cursor, the only thing that got wrong was the border color. Everything else was perfect.

Dylan [00:23:39]: I’m glad to hear that. That’s awesome. I was very impressed. And to be clear, some of the time it does just work. Exactly. And it’s magical. It’s magical. And by the way, I’ll point out, that’s not like a comment on, oh, front-end engineering is dead. There’s so much more to front-end engineering than just that translation step. That’s kind of like the most mechanical part. And actually thinking through all the states, all the intended behavior, and how to make the design truly come to life, that is a lot of the interesting work. And I’m excited for how interfaces will get much more rich and much more interesting as we make it so that they’re easier to go from design to code in that first place.

Alessio [00:24:20]: Because that’s just the first state. Right. Yeah, when we had Greg Brockman, we talked about these purple and blue gradients kind of taking over the web because of all this training data. Do you feel a sense of responsibility in a way in Figma Make also to set the new standard for what things should look like? And how much do you think about making that explicit for the user right on? It’s like, hey, I’m actually not going to... You need to make some aesthetic choices early on. Yeah.

Dylan [00:24:45]: I feel very strongly that across the Figma platform, we should do our best to find ways to help people explore more of the space of aesthetic rather than impose a personal viewpoint on aesthetic. That’s a hard problem. But if we can accomplish that, I get very excited. Because if you can actually open that space up more and figure out how it applies to software, to product design, then not only could we... We’d be in a place where we help you generate high-quality visual output, whether you’re a trained designer or not, but also we could get to the place where you could be nudged into or nudge yourself into different directions that are underexplored relative to the design community and design history. As you know, it’s like there’s an ability to interpolate between different styles, different ideas, and AI can help you do that. But also the designer can then take that so much further. Yeah, I think regurgitating the median website is like, you know, maybe that’s where a lot of us are today. But where we need to get to is one is a place of really pulling out new styles. And I think overall, the other thing I’d say is you look back at the Flash era of the web. You know, we both grew up through that. And it was maybe not always high quality, but dynamic. Exciting. Fun. You know, this was an era where experimentation was happening. And then we, you know, at some point it was like, okay, Steve Jobs gets up on stage and goes, Flash is dead. You know, here’s my new world. And, you know, brief skeuomorphic phase, then Swiss Minimalism. And we’ve all been Swiss Minimalism for quite a long time now. And I just think that, again, going back to, okay, more software created than ever. What needs to happen? Well, designers need to push us forward. And that’s going to mean an exploration, an explosion of creativity and so many different visual styles that will be explored. So much more dynamism in interfaces and new patterns emerging. Especially as you start thinking about what are all the screen targets we’re going to have. Those are going to explode too. And states, you know, all the different surfaces that will be created. And designers will have to think through systematically. That’s a big challenge. A big opportunity as well. Yeah.

Alessio [00:27:13]: I started, you know, Kernel Labs a month and a half ago. And James, our designer, I’ve been working with him on our initial landing page and our product design. And he’s almost like my Latent Space shepherd. You know, it’s like, even if I had, even if Figma was AGI, I still wouldn’t necessarily know what to ask for. You know, I think that’s like really what people a lot of time get wrong, which is like software. It’s like the same thing. It’s like you could give a software AGI to anybody and like doesn’t mean they could build great software. Correct. You know, I think to me, that’s the most exciting thing. But what I really like is then the ability now that I have to like reuse this across surfaces in a way that wasn’t possible before. You know, because I can create ideally in Figma, not my design system, but now you have make to generate all new types of products. I have the NCP. You also have, you know, slides. You have like different products on the other end of the spectrum. And again, going back to being like the context repository of aesthetics. I know that as long as I have, you know, the context from Figma. Whatever the AGI or whatever I’m talking to is going to generate, it’s going to have some sort of like rooting and like what I think looks good. Yes. I think some people will have it personally. I think some people should have some sort of like personal Figma almost. Where it’s like, you know, when you’re generating images on ChatGPT or Midjourney, you should have some sort of like aesthetics to draw from. How do you think about that evolution? Because you’re going from a world where like only designers work on your product and now it becomes a core part of like a lot more constituents. Yeah.

Dylan [00:28:39]: I think in a world where design is the way you win, it’s only natural that we need to get more people involved in the design process. That is not going to diminish the role of designers. In fact, I think it expands the role of designers because then you have to shepherd people through the design process and help them go from, okay. I mean, it’s kind of a journey you go on, right? Like not even being aware of design. Yeah. You know, kind of like blindly going through the world to, oh man, aesthetics, they matter. To, okay, can we make it pop? Can we make it cool? And then people start to actually think about it. Well, wait a second. Like I’m looking at one screen. What’s the actual experience here? What is the entire flow? And then it’s, okay, well, let’s take mental models of how we can think about this experience and consider it in different ways. What are the potential different paths? Metaphors and experiences that we can create here? And what are the abstractions that matter? And then from there, it’s like, okay, wait a second. Well, this all exists in the context of like our brand, the greater culture of the moment and business constraints and all sorts of other things you might be optimizing for. And I think more people coming to the design process. That can help add in context as well. And there’s no reason why someone who’s outside of design or doesn’t call themselves a designer, whatever they identify as engineer, product manager, CEO, whatever. Everyone should be able to come in and say, okay, here’s an idea. And the idea hopefully could be parsable in high fidelity to the standards of at least the design system and consistent. Yeah. So it’s not distracting because an idea should be evaluated on some merits. But then I think from there, the actual exploration and making that great, that is a hard design task still.

Alessio [00:30:47]: So how do we lower the floor for everyone coming in, but also raise the ceiling, make it so designers can do even more and produce even greater work? I think like, I mean, obviously, you know, it’s cliche, like everything is changing, right? But I think there’s just like a fundamental shift in both. So how people perceive software. I think, you know, Sam tweeted at the moment, tweeted about the fast fashion era of SaaS. But there’s also like a negative connotation to fast fashion. But I feel like in software, it’s like, man, if you can get the software that you need at any moment, that’s not a cheap thing. That’s like an expensive thing that is now being made cheap. And it’s like still high value. And I’m curious like how different products are going to drive that. Even though it feels like, hey, I just created this for you. Very quickly. But like there was so much that went into that, you know, that is like maybe sometimes undervalued, you know? So I’m curious, like if there’s something that you think about where, okay, in a way people should come to Figma and do a lot of work. But maybe in a way we can kind of help you from like all the work you’ve done in the past, kind of like come to the right result much faster in the future. Yeah.

Dylan [00:31:56]: Well, I think especially in the context of teams where there’s, you know, they’ve done a lot of work in Figma. Of course, there are patterns that with that team’s consent, you can tap into and figure out how to help improve outputs. But I do have a little skepticism about the like fast fashion interpretation. I just think that there’s where we’re currently at at least with the models. And, you know, of course, we’re on this trajectory, whether we’re on an S-curve or an exponential or it’s an S-curve that will turn into an exponential. I don’t know. Maybe you’ve got a point of view. Sure. But I just kind of like, okay, I’m excited for the ride. And as long as… I’m here today. Well, my mental model and strategy is just like your strategy should always be okay. Assume AI models get better and make sure that makes Figma better. As long as I believe that’s true, I’m happy. If not, change strategy. That’s the algorithm, you know. But wherever you’re at, I think that in terms of that interpretation of where models are going, I don’t think the world is in a place today where those models are going. I think the fast fashion era is here. And I also think that so much of designing software is doing it in a way where many people can use it. Like, it’s rare, I think, that you’ve got an individual piece of software that is truly just for you. I think it’s awesome that more people are exploring their ideas and creating software and tools for them. But then the next step, if they want to go further, is, okay, how do I make this good for other people, too? And in a setting where people are trying to learn software, most people learn it from other people. So now you’re in the same place you were before. You had a piece of software. You made it just for you. You decided it actually supplies other people the problem that you solved. Now you’ve got to have something that is probably consistent enough to actually share with others so they can learn it. And it gets adopted. And so I don’t know. I think it’s like, yes, more people will create software. That’s awesome. But also, I’m not sure that software will just be, like, disposable. And if you look at the way that people work, whether it be with Cloud Code or Cursor or Warp or whatever, so much right now is, like you said, you need to have some expertise about how software is built. That lets you discretize the task, just like you would to an intern. And, you know, maybe it goes beyond intern level. But still, I’m not saying, okay, go build Figma. And you, agent, are just going to go figure out all the complexities of Figma. I think that’s just not something I see happening in any near-term future. Even as longer-running agents start to occur and we’ve got better capabilities, that’s a long ways out. Now, maybe that’s a high bar. But you look at the actual workflows that happen with, you know, the very big SaaS applications. Let’s consider Workday, for example, or Salesforce. A lot of CIOs would, you know, love to go, okay, yeah, I’ve vibe-coded Workday. And I just saved my company all this money or whatever. But, okay, you actually peek under the hood of a Workday or a Rippling. These are very complex pieces of software that have accounted for every edge case that you can run into as you’re thinking about your HRIS. And the platform of data that you can tap into. Yeah. And then go build out from there into different workflows. And they’ve done that through, you know, over, in Workday’s case, decades. In Rippling’s case, you know, not quite a decade, but also a lot of prior knowledge about what needs there are in the market. And very intentionally built. So I’m skeptical that, like, without that knowledge of the workflows that people will encounter, people will actually make something that can scale. I think you run into the same problems that you’ve run into all along. And then it’s a loop. Maybe that loop goes a little faster. But it’s not, like, just going to replace.

Alessio [00:36:04]: I think that’s the bull case for software still being helpful post-AGI, whatever that means. Because in a way, you still need to prompt the AGI. And I think you’re going to end up having these interfaces that, again, just like you’re going to help people explore the Latent Space in design, there’s going to be a way for interfaces to, like, compress the way that information gets passed through these systems. I think in data analysis, you’re kind of seeing a lot of the applications kind of going away in a way because the models are, like, so good at it and it’s so natural to do on a conversational level. But again, sometimes it’s like, well, how do you give it the right data? How do you ask for the right type of charts? How do you ask for, like, the right follow-up questions? I think there’s, like, a good question of at what point is a software a piece of software, at which point it’s just like, again, a Latent Space guide that just helps you project your interest into the model. Yeah.

Dylan [00:36:56]: And it’s interesting. It’s like if you look at data analysis, maybe break it up into a few things that have to go right, you need to have, first of all, trust that the right queries are being written in the correct way. And that’s a lot of trust because if you get that wrong, you know, you have a bad shaky foundation for the rest of your experience. And there’s, okay, what’s the next query? I’d probably be more bullish about AI predicting the next query or a follow-up than I am about, like, you know, 100% rate on the query being constructed correctly. I think if you were to show people some prompts around here, example, next queries you might want to run, that might spark other ideas they have for follow-up questions that further their analysis. But there’s also, like, how do you display the data and the visualization itself? Yeah, there’s canonical visualizations we’re all used to. But I think visualization is fascinating. And we’ve only scraped the surface in terms of how we can visualize data. I think that especially to get to larger data sets, more complexity, and you’re really trying to communicate data to people, that is one of the most interesting design problems out there. Like, how do you communicate how much money in the budget of the federal government is spent where? People have tried so many times. I’ve never seen something that is clear and actually gives you any sense of scale that you can relate to, you know. And how do you communicate? How do you communicate the data inherent in biology to a layman person who hasn’t studied biology? Like, again, people have tried. It’s a very hard problem. And maybe you break into sub-problems, but still, there’s so much to push on there. Yeah.

Alessio [00:38:42]: Yeah, exactly. There’s almost, like, the two ways. So, like, me, how do I communicate to the model what I need? And then there’s also the other side, which is, like, the models have, like, so much imbued into them that we need to get out of it. That we don’t quite know how to do. And, like, one part is, like, information communication. You know, one part is, like, once I prompt it, how do I get the response in a way that I really parse? I think, yeah, that just, like, I don’t know. It’s been breaking my brain for the last few months just thinking about what will feel like software, what will feel like a conversation. And, like, I know that, obviously, OpenAI, you know, has, like, this big goal of, like, kind of being your companion and, like, you have the voice mode and whatnot. But at some point, you just need something that is beyond that. You need something beyond component rendered inside a chat interface, you know? And it’s hard to figure out, especially when you think about, I know you do a lot of angel investing. So, I’m also curious about how you think about startups and, like, what kind of products are, like, now possible that maybe weren’t before? Like, what products people should stop pursuing because you think will be a part of the models?

Dylan [00:39:43]: First of all, I’m hesitant to give advice on that because, first of all, I think that the idea that all software will or lots of software will exist. In a session with an LLM or any model, I think that’s a little overblown. There’s so many different ways this could work out, whether it be, you know, kind of a back and forth of the model as an origin of a request. And then you go elsewhere to the models embedded in software. But you still have a destination you go to first. It’s not the model. And we can think of many other ways, too. I think that that’s maybe a bit of a shaky assumption. And then also, I think that people, it’s often the case that you’ve got some space that, like, a thousand people are starting a thousand companies in. And everyone’s going, oh, don’t go there. It’s too crowded. But then one person comes up with some really new, clever idea. And it’s a totally different take. And they propel from there. So, I never try to say don’t do something to an entrepreneur because somebody that’s listening to this podcast is going to have some ego. They’re going to have some unique insight in some space that both of us think is really dumb to work in. But then they’ll be the next, like, you know, trillion-dollar company. So, and, you know, of course, like, whoever that is, you know, let me know. Right, yeah. Let us be a part of it. Let us both know. But, yeah, I think there is some amount of memetics around, like, people see other people doing something and they follow along. And you have to have a unique insight if you’re starting a company or working on a product. It usually should be something that’s unpopular. Right. And this is just cliche advice at this point. But, like, I think there’s something deep to internalize there about the kind of contrarian nature. I’m going more tea language now. But I think he’s basically right about this. I mean, as in if you’re investing in something, unless you’re just going after momentum, which a lot of people do. But otherwise, you need to have some point of view that, like, a lot of people would just blanket disagree with. And it should be scary to you if you’re investing in something and you tell your friend about it and you say, here’s my point of view on why this is really cool. And your friend is like, oh, yeah, I totally agree. It makes complete sense. That should be a warning sign if you survey people and they’re all saying the same thing about that. Yeah.

Alessio [00:42:12]: What have you learned about yourself during the TO fellowship? What are, like, things that you change about how you approach, yeah, just life, thinking, learning?

Dylan [00:42:20]: think whether it be, like, that interaction with Chris where I look back and go, man, I maybe dismissed that one too soon, you know, and then learned over time, thankfully. Or another example is I think in 2013, there was, like, a Bitcoin hype cycle. You know, Bitcoin went to $1,000 or something. And, like, half the TO fellows at the time were really excited about Bitcoin. I’m just like, these idiots, like, how do you short this thing? And, like, that was my default reaction. And I think that the overall meta lesson that I’ve learned over not just the TO fellowship but just being around tech for a while now because I was paying attention even as a kid. I was in some commercials that were, like, for, you know, Microsoft and eToys, for example. And so then I’m starting to track, like, what’s this dot-com bubble? And wait, why is that coming? Why, like, am I not getting residual checks anymore? You know, that became, from a monetary standpoint, I’m like, I’m going to read the newspaper. And then, you know, working in high school at O’Reilly Media was a great point to get exposure to some of the starts of cycles, TO fellowship. And the meta lesson I think that I’ve learned is don’t look for reasons why things are not going to work. That’s important too, but it’s not the place to start from. The place to start from is, like, what could this be? How big could this be? How important could this be for society? And let yourself imagine, dream. And then go and think about all the ways it’s not going to work so you can mitigate each one. But, like, start with the dream. And I think if you start there, it’s just a better default position to go from.

Alessio [00:44:05]: Yeah, I’m really worried about the X algorithm and what it has done to optimism because it’s so easy to get likes just being negative about things.

Dylan [00:44:14]: I do think the X one seems to have changed a little bit recently.

Alessio [00:44:17]: Yeah, maybe Nikita’s in there tweaking things.

Dylan [00:44:20]: Yeah, I’m thankful for that. But, yeah, I do think algo feeds reward controversy and being negative is a way to get controversy. But also, I don’t know, I’m default optimistic. I feel like society just builds antibodies to different things over time. I mean, remember when we were all worried about, like, oh, my God, everyone’s going to be a zombie playing Farmville all day? Right. Like, well, here we are, you know. Yeah. Some people still play Farmville, but, like, it’s. Right, mostly. I don’t know anyone that does that, like, you know, all day long. And most of us kind of forgot about, you know, the social gaming era.

Alessio [00:44:58]: I know. So. When I drive by the Zynga building, I’m always like, I remember, like, in the days. Yeah. How do you use X? Because, I mean, you were famously on IPO Day responding to product feedback on X. What’s your routine for staying on top of that?

Dylan [00:45:13]: Oh, I mean, I try to just, like, search for Figma a lot. And see what people are saying. But also, I’ve, like, trained my algo feed to show me a lot of stuff that’s relevant to Figma. Yeah. And there were ways. I don’t know if they still are as powerful signals with whatever algorithm shifts have happened. But it’s, like, you kind of find out what signals matter, you know. Like, ironically. Right. Not interested in this post. Seems to not do anything. You know. A like or a bookmark. I’m not sure how much that matters. But copy link. Turns out that really matters as a signal. Or at least it did. So, you know, I wasn’t always sharing the link, but I’d copy it whenever I saw something I wanted a signal boost in my feed. It’s like, okay, the more you learn as X for the algo feed, the better you can train it, the better you can make it useful. And then, you know, I think feedback across any surface, not just social media, but support, sales conversations, conversations in our community. We gather people together and then just talking to folks. Research both qualitative and quantitative. These are extremely useful signals for our team. And I think of intuition as like a hypothesis generator that you have to test the hypotheses. So, using feedback to be part of that test is important. But also, I’d always rather give feedback to the Figma team by surfacing the voice of a user rather than being like, I have this point of view. Right. You know, I’d do the latter as well. But the former is my preferred method. And so, I’d much rather champion user feedback or a user bug report. Right. So, if I have a feature request and then dive in with that person, then, you know, just have it come from me. And the other thing is I’m always looking for those visionary users who are a step ahead of everybody else. They know just kind of intuitively what is needed. And I think that when you can find them and separate them out, that’s a signal that is just amazing to get. I remember early days of Figma. There was this one user test that I did. I literally brought a bottle of wine, too, because Figma was so slow yet at that point. I mean, like, to complete the user test, I knew it would take hours. So, it was kind of a tough one to administer. We went through the bottle of wine during the user test. And the person that we were doing the user test with, this guy named Payam, who’s an amazing designer, then was working at Coursera. And the next day, he followed up with, like, this super long doc. I mean, it was eight to 10 pages. And it laid out basically a lot of what ended up being our roadmap. Not like we literally follow the doc, but I look back and compare contrast. And yeah, he’s an example of someone who was a visionary user. And for a person like that, there’s a lot of people that will give you more local feedback. But some people can see the big vision, too. And that’s always really exciting because it’s validation for you and the team about

Alessio [00:48:10]: where you should go, but also a source of new ideas and insights as well. How did you think about hiring and building the team back then? Because I remember I was working at a YC company and we were all on Sketch. Kickback. Okay, cool. And we were all on Sketch, right? And so, I think a lot of people are maybe like, well, why isn’t Figma just going to be like Sketch and blah, blah, blah? Yeah. How did you figure out who are the right people to bring on the mission? Because I think the same thing is happening in AI, which is kind of like the, you know, the meme of what should be built. And then maybe there’s like some more missionary people. What were some things that you think people should take on early stage? Early stage recruiting, especially.

Dylan [00:48:46]: Early stage recruiting is so hard. So, first of all, just don’t give up. Right. That’s my first piece of advice. Second piece of advice is just think long term. You know, there are folks that I talked with in the first year or two of Figma, and they didn’t join until like year five, year six. But those relationships, it’s amazing how if you’re consistent and just spending time together with people you like, how they’re going to get along. And then eventually it turns into something that could be they join a company or actually just throw a friend outside the company, but someone that inspires you. And that’s great too. But yeah, I think taking the long view is super important. Of course you need conversion today, you got to hire. And I think the kind reality of being early stage and having a lot of risk is you have a natural filtering function. Right. Like, only the true believers are going to get on board. And I’m a fan of just not selling too hard. It’s like, make sure people understand what’s going to go on and what’s going to happen and where you’re pushing and what you’re going to do. But like, if someone needs to be sold so hard, it’s usually a sign they’re not going to stick around. Yeah, I think you have to have a really good process. Best recruiting advice I got in the early days, yeah, I told John Doerr one day, he said that, you know, I was having a lot of problems with recruiting. I wasn’t very good at it. And he’s like, well, do you wake up in the morning and it’s the first thing you think about recruiting? I’m like, well, no, I’m thinking about like coffee, like, okay, well then, you know, it’s like mid-morning. Like, are you thinking about recruiting? I’m like, no, I’m probably thinking about like what snack I’m going to have. I’m just like, okay, it’s lunch, are you thinking about recruiting? I’m like, no, I’m probably thinking about like what emails I got to do. Okay. It’s like last thing, the last part of the day, you’re about to go to bed. Right. I’m like, am I recruiting? I’m like, no, definitely not. I’m like tired. He’s like, well, just like, if you’re thinking about recruiting and all these moments where you just have a moment, have a second to pause and you’re actually on it, then it’ll fix itself. And the way that I think can manifest as a process is you just basically make a spreadsheet of here’s my funnel and you obsessively look at it all the time and go, okay, like, how do I make sure that this funnel? Uh, keeps going. Just like you would with a sales, if you’re a salesperson, you have to continue to feed the funnel. You have to move people through it. And if you’re not doing that, you’re not recruiting. So yeah, you have to be very disciplined, which is something that, you know, I’m, I have to push myself to, I like to kind of be in the cloud. So

Alessio [00:51:30]: I definitely like to have coffee. Um, how has that changed now that you have like, you know, public company that you run obviously would make, I’m sure you have to build a new team to kind of lead that. Yeah. How has that changed? And also like, you know, uh, this is a great call for recruiting, uh, for engineers listening. What are like the type of people that succeed at Figma today?

Dylan [00:51:51]: One thing that’s been interesting is that, uh, yes, of course we’ve hired amazing researchers who are pushing the boundaries and thinking, you know, in a way that’s, you know, on its own life cycle in terms of not as tied to, okay, we have an explicit date we’re trying to release on because that’s just not how research works. But what I’ve found is, um, very good. Yeah. I mean, you know, we’ve had engineers who are just more full stack and oriented towards learning new things. They can be quite successful on AI products. There’s some reorientation they might need to do. They might need to, of course, learn new skills, just like designers need to learn new skills. But in general, we’re looking for smart high agency people who have product sense, who care about design, who see the world the way that we do. And, and want to learn skills and keep growing and work on hard problems. I think that’s the filter that’s always been the case for Figma.

Alessio [00:52:48]: And, um, if that resonates with people, then yeah, please apply. One question I have from Zach from Warp was how do you position Figma make in like the universe of this like prompt to app, prompt to creation? Like how should people think about how you fit? Do they feel like competitors to you? Do this make just feel like an extension of the design team? What is like the core? What’s the competitor universe for you?

Dylan [00:53:11]: You know, it’s depends on how you define it, right? Like I think, um, engineers feel much more comfortable in ID than non-engineers. And as we get to more agentic environments, perhaps that’s a different vibe, but still, I think most people feel like, uh, an ID or cod code, if they’re not engineers, they feel like that’s not made for them. Yeah. And I think that with the platform we have and the visual metaphor, the opportunity for free form exploration, ideation on an infinite canvas and being able to try out lots of things and then also see the big picture of what are the different paths I can go down. That is a metaphor that I think works for a lot more people. And in that sense, as we tie, make in even further than just, okay, I can copy a state, input, and Figma design. That’s what we launched today, but there’s so much further we can go. And the further you go, the more you’re able to then, I think, bring more people into this sort of surface. So in that case, you know, it’s, it’s just like, we’re trying to race against ourselves. I think, um, you know, if you evaluate Make as it launched, we raced towards launch, you know, as prompt to code and not a lot around that, not a lot of integration with the Figma platform. Then sure. There’s a million other tools and more coming every day that you can evaluate it against. But I don’t know if that’s the right way for us to think about it. I was, um, listening to the Brett episode you did, and he had this one line that’s just stuck with me, which is, he was talking about, uh, you know, do you want to resell coffee beans as roasted coffee beans? Or do you want to like go make the amazing, like special latte, obviously want to make a special latte. Yeah. Everyone does. But like, how can you make something that is, uh, unique in place, the needs of designers and does that in a way where we can really bring advantage to people. And I think there’s so much we can do there. So many different tiers that are coming that I’m just really excited about. Yeah.

Alessio [00:55:19]: Nice. I know we don’t have too much time left. I want to just talk about some outside of tech. So you always have a CryptoPunk as your Twitter PFP. How do you think about it? It’s actually a chain runner, not a CryptoPunk. See? Okay. Uh, we’ll, we’ll go through NFT education later. How do you think about in the digital world, especially like you have, you’re gonna have this huge divide between scarcity and like, you know, abundance, right? How do you feel about the future of like these digital collectibles and like communities and like, yeah, how that fits into the universe of like, Hey, you can generate anything at any time. You know, Enzo Ferrari used to say a Ferrari can never be readily available to be desired. I’m curious how you think about the distribution of things. People will see on the internet between the super niche tailored, just created for you and like these kind of like iconic cultural properties.

Dylan [00:56:09]: The paradox of like digital scarcity is what made me excited about NFTs before they’re called NFTs back in 2017, whatever it was. And I think that, uh, not everyone is like, has that collector gene, but some do. And, um, for those that do, whether they’re, whoever it is, they’re collecting digital items. They’re scarce and people will enjoy getting into whatever collector aspect they want there. But I think in general, I’ve, I’ve found myself wanting to distance a bit from the NFT space. It’s like kind of interesting and actually has some parallels, I think, to AI and I got in it so early and that point it was like this like just niche community on the internet of weird people that thought digital stuff could be scarce and might, you might want to collect it and pay real money. Yeah. But it was not expensive. And so anyone, for example, in the States, like could be part of that. And you really, it was gated more on just like, do you know about it and do you get the idea of it? And is that idea exciting to you or is it repel you? And, um, then I, like over the next three or four years, it went from this like very idealistic would think about what the future of creation and digital items and scarcity could be. So to get rich quick scams and just sort of this overall vibe of like flipping stuff and trying to make money and I don’t know, it just the, the whole meta of it changed. And I, that’s when I peaced out, you know, I, I realized at some point I was like, oh, I get excited about collecting some NFT project because I think that it’s the art is cool. I think that the creators are awesome and there’s some intention behind the work that’s And then, you know, if I talk about it online, like the project might do worse. It might attract people to buy it and folks can apparently, I learned, do scams where they basically pump and dump and create, you know, trading behaviors that are no good. And so I stopped talking about things I’m excited about for the NFT space. The parallel that I think is kind of interesting is, you know, you compare AI to that. And it’s like, there’s been a long era of people that I think are very on mission and thinking about the big picture, the risks, the opportunities, the possibilities.

Dylan [00:58:42]: And that’s kind of meeting in this moment, the get rich quick. You know, if you look on YouTube, there’s a lot of people making videos about like, okay, how do you use AI to make business?

Alessio [00:58:51]: Passive income. And I’m not trying to dismiss that because great, if you can make money using AI, that’s great for you. And some people certainly will. But I think there’s too much just like, do it because it’s going to make you some money, energy in the space right now. That makes me like a little bit nervous, having been through that NFT cycle and seeing where it ended up. That has been on, you know, I own a card store in San Carlos. So I do like Magic the Gathering.

Dylan [00:59:21]: And there’s similar thing happening where like, you know, there’s a lot of speculation just because everything is the gathering, like super cool now.

Alessio [00:59:28]: It is. Oh, fuck. Yeah. Well, I’ve been waiting for this moment. Let’s go. We’ll do it. We’ll do a Magic the Gathering event.

Dylan [00:59:34]: Draft Night. I’m down. Nice. We used to have it in the early days of Figma. We used to do Draft Nights.

Alessio [00:59:39]: Nice. Yeah. What sets were coming out then? Do you remember?

Dylan [00:59:42]: I don’t remember.

Alessio [00:59:43]: Okay. Yeah. Sorry.

Dylan [00:59:44]: There’s a guy named Andrew on our team. And he wowed me so much. With his expansive encyclopedic knowledge of Magic the Gathering that I was like, wait a second, like, can we move you from support to product education? And then he killed it at product education because he has just as encyclopedic knowledge of Figma. And but yeah, it was like basically Magic the Gathering Draft Night that gave me the confidence inside of, oh, wow, like these skills are transferable. So that’s funny.

Alessio [01:00:17]: Magic the Gathering, career opportunity. Exactly. You know, I should go around. I’m going to play in the regional championship for the Americas in November. I should just go around and say, okay, you should come work at this. You’re going to the regional championship?

Dylan [01:00:27]: Yeah.

Alessio [01:00:28]: I see you’re like really hardcore. So I think to me, that’s like the best way to like disconnect because you have to be so focused on the game that like you’re not actually thinking about things. But there’s kind of like obviously the collectible side, but there’s still at the core, like a community, let’s come together at the store, hang out, play games. And I hope that like, that’s what we’ll see more out of AI, which is like enabling more. Me too. Which is like enabling more. More of these like small communities locally to like, you know, have more entertainment and like support themselves in a way that doesn’t have to be, oh, is this going to make money? Like, is this going to be profitable?

Dylan [01:01:01]: Yeah, I think the more you can go from a mode of like, I go on social media app of choice and mindlessly flip through my algo feed to I’m going and making things like that is good. We want to move consumption behavior to creation behavior. And yeah, I think that will happen. I just a little nervous about the get rich quick fives. Right. Yeah. Awesome.

Alessio [01:01:25]: Dylan, we’ll have you for draft night at the new Kernel Space. Looking forward to it. But thanks so much for the time. Thank you. Thanks for having me.