AI spent the last few years competing on intelligence. Now the next major AI race is about speed.
As AI shifts from simple chatbots to agents that reason, write code, call tools, search the web, and coordinate other agents, latency compounds. A task that requires dozens or hundreds of sequential model calls can quickly turn seconds into minutes — or even hours. That makes inference speed a fundamentally different problem in the agentic AI era.
In this episode of So What About AI Agents?, Philippe Trounev sits down with Vasanth Mohan of SambaNova Systems to unpack what actually makes AI faster — from model architecture and memory bandwidth to batching, specialized accelerators, and next-generation inference hardware.
Vasanth explains the two major stages of inference — prefill and decode — and why they create very different hardware bottlenecks. They discuss operator fusion, parallelization across chips, memory bandwidth, and why reducing latency often comes with a significant cost tradeoff.
They also explore why coding agents are one of the clearest use cases for premium inference, how AI providers may eventually offer commodity, premium, and ultra-fast tiers of tokens, and why businesses willing to pay for speed may gain a meaningful productivity advantage.
Vasanth also shares early performance figures for SambaNova's upcoming SN50 architecture, including benchmark results around 800 tokens per second on a large model, compared with roughly 300–400 tokens per second on GPUs in the cited comparison.
We also get into a slightly crazier question: are AI agents already helping engineers design the next generation of AI hardware? The answer is increasingly yes — although humans are still very much in the loop.
In this episode:
- Why AI agents make inference speed dramatically more important
- Why sequential agent workflows create a latency bottleneck
- Prefill vs. decode explained
- GPUs vs. specialized AI accelerators
- The relationship between speed, batching, throughput, and cost
- Why memory bandwidth matters for large AI models
- What SambaNova's SN40 and SN50 architectures are designed to solve
- 800-token-per-second AI inference
- Why coding agents benefit so much from faster models
- Commodity vs. premium vs. ultra-fast AI inference
- Whether faster AI becomes a competitive advantage
- How AI agents are already being used in hardware engineering
- Why AI infrastructure may become just as important as the models themselves
Chapters
00:00 — The AI race is shifting from intelligence to speed
00:39 — Why AI agents suddenly need faster inference
02:38 — What actually makes an AI model run faster?
05:30 — When does ultra-fast inference matter?
08:08 — Does model architecture determine inference speed?
10:05 — How do you optimize AI from model to hardware?
12:44 — How fast can AI inference actually get?
15:28 — The sequential latency problem with AI agents
16:39 — Where faster AI creates the most value
18:30 — Will premium AI inference become a competitive advantage?
20:11 — What does AI inference actually cost?
23:10 — How many tokens can AI hardware generate?
24:46 — Building private AI infrastructure
28:02 — How much faster can AI eventually become?
30:14 — Are AI agents already designing AI hardware?
32:50 — Why today's “fast” AI will eventually feel slow
33:34 — What happens next in AI infrastructure
