Beyond The Pilot: Enterprise AI in Action

AI agents can now be weaponized to run elite-level cyberattacks at scale — and your perimeter won't stop them. Box CTO Ben Kus lays out exactly how enterprises defend against agentic threats without locking down the tools their teams actually need.

Ben Kus spent years at IBM's BigFix patching vulnerabilities at enterprise scale. Now at Box — where 95% of the codebase is permissions, security, and compliance — he's dealing with a new threat model: cheap, scalable AI agents that give any attacker access to elite hacking capabilities previously limited to nation-state actors. Only 14% of companies in VentureBeat's September survey said they could shut down a misbehaving AI agent within minutes. Ben explains why that gap is dangerous and what the architecture of a real defense looks like.

The conversation gets technical fast. Ben walks through the "lethal trifecta" (untrusted input + sensitive data access + outside communication = critical risk), how Box uses action guardrails to sandbox agents even if they're compromised, why 44% of enterprise agents running on shared human credentials is a ticking clock, and how newer Anthropic models have dropped prompt injection success rates from 60% to roughly 2%.

The second half shifts to retrieval. Ben breaks down why standard RAG fails at enterprise scale — permissions gaps, vector space degradation, poor lexical matching — and what Box's context retrieval system (hybrid embeddings + reciprocal rank fusion + agentic search) actually delivers: 90% token cost reduction and 4–5x speed improvement over naive model-provider retrieval. He also addresses the harness question directly: 59% of companies that swapped models in production saw performance drop, and the reason is usually the harness, not the model.

🎙️ GUEST: Ben Kus | CTO, Box

🎙️ HOSTS: Matt Marshall | VentureBeat, Sam Witteveen | VentureBeat

---

**CHAPTERS**

00:00 Intro — AI agents as elite-scale attack tools

02:00 Episode overview: Box CTO Ben Kus on agentic security and retrieval

03:00 What's actually changed in enterprise security: speed, severity, AI-assisted vulnerability discovery

04:20 How AI finds multi-domain vulnerabilities humans miss (PHP, font exploits)

05:00 When agents become attackers: democratizing elite hacking

06:20 Why perimeter defense is dead and what layered security actually looks like

08:00 VentureBeat survey: only 14% of companies can kill a misbehaving agent in minutes

09:20 Agent identity and credential sharing: when it's acceptable and when it's not

10:40 Misaligned agents: the agent-with-your-identity threat model

12:20 The lethal trifecta: untrusted input + sensitive data + outside communication

13:00 How Anthropic models dropped prompt injection rates from 60% to 2%

13:20 Box's approach: action guardrails, sandboxing, MCP server restrictions

14:00 Prompt injection via documents: how Box cleanses untrusted content

15:20 Action Guardrails feature: containing damage when agents are compromised

16:20 The SaaSpocalypse question: will Box become just a connector?

17:00 Why systems of record survive the agentic shift (headless platforms, MCP)

18:40 Agent-to-agent delegation: when Claude routes work to the Box agent

21:20 Why enterprise search has always been broken — and what's changing

22:40 Vector embeddings, semantic search, and the limits of pure RAG

24:00 Reciprocal rank fusion: combining lexical and vector search at scale

25:00 Box's "librarian" model: returning pages, not books

26:20 Context retrieval in Box Works: 90% token reduction, 4–5x speed improvement

29:40 Why throwing away shipped work every 3–6 months is now a core skill

31:00 59% of companies saw performance drop when swapping models — why the harness matters more than the model

33:00 Custom vs. generic harnesses: how the answer has changed three times in one year

33:40 Closing thoughts

---

Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat

Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239

Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4

Website: https://venturebeat.com

LinkedIn: https://www.linkedin.com/company/venturebeat

Newsletter: https://venturebeat.com/newsletters

#EnterpriseAI #AIAgents #CyberSecurity #RAG #AIInfrastructure

Learn more about your ad choices. Visit megaphone.fm/adchoices

Home Depot cut delivery failure rates from 5.3% to 2.8% in Chicago using a purpose-built agentic system — and it's protecting 1.5 million at-risk deliveries a year across weather, geospatial, and customer preference signals.

Home Depot CIO Angie Brown breaks down exactly how the sixth-largest U.S. e-commerce operation ($25B+ in online sales, 2,300+ stores) built its production AI stack from the ground up. The architecture is a layered agent hierarchy: a thin client planner routes to domain agents (selling, post-purchase, delivery), which hand off to fine-grained sub-agents by category — paint, power tools, configurables. Underneath sits a knowledge foundation built on Google BigQuery, hybrid search, unstructured vector databases, and a data mesh, all wrapped with agent identity management, security guardrails, and safety evals.

On the model side, Home Depot runs Google ADK as its primary orchestration layer with Azure OpenAI held as an active counterweight to evaluate output quality — and Claude for code generation with OpenAI models used for code review. The Magic Forge internal platform handles their most compute-intensive workloads. Multi-model tension is a deliberate architectural choice, not an accident.

Two production results stand out beyond delivery: a conversational IVR (Google NGA) now live in ~1,000 stores is handling 3x more customer questions without human transfer, and an AI-powered blueprint takeoff workflow for contractors turned quote turnaround from one-plus weeks down to 24–48 hours. Angie also covers where the architecture still has rough edges — deterministic escalation rules being layered back onto the conversational IVR — and what's next: per-SKU assortment optimization and agentic planogram generation across 30–40K SKUs per store.

🎙️ GUEST: Angie Brown | CIO, The Home Depot

🎙️ HOSTS: Matt Marshall | VentureBeat, Sam Witteveen | VentureBeat

---

00:00 Intro & delivery failure rate results

00:57 Home Depot's technology footprint: 2,300 stores + e-commerce scale

03:20 Magic Apron walkthrough: AI-guided product discovery online

06:00 Fulfillment AI: how estimated delivery dates are computed in real time

08:20 Order Intelligence: weather, geospatial, and customer preference signals

10:40 Agent architecture deep dive: client planner, domain agents, sub-agents

13:20 Meta-agent question: how context inference replaces a top-level orchestrator

15:00 Full AI stack: Google ADK, MCP, LLM eval, prompt management, vector DBs

16:20 Multi-model strategy: Google vs. Azure OpenAI vs. Claude — how they decide

19:00 Magic Apron consumer rollout: what worked and what didn't

19:20 B2B pivot: List Builder AI and blueprint takeoff for contractors (24–48hr quotes)

22:20 How the agent framework has evolved: less summarization, more model intelligence

24:20 IVR in 1,000 stores: conversational AI replacing tree-based phone menus

28:00 Associate augmentation: Magic Apron in-store mode and plumbing aisle story

30:20 Customer adoption of AI IVR: what the call recordings actually reveal

32:20 Deterministic escalation controls layered onto conversational AI

32:40 LLM-referred traffic and the future of agentic commerce

37:00 What's next: agentic assortment optimization and planogram generation at scale

39:00 Outro

---

Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat

Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239

Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4

Website: https://venturebeat.com

LinkedIn: https://www.linkedin.com/company/venturebeat

Newsletter: https://venturebeat.com/newsletters

#EnterpriseAI #AIAgents #RetailAI #LLMInfrastructure #AgenticAI

Learn more about your ad choices. Visit megaphone.fm/adchoices

A $600 day of AI coding spend sounds alarming — until you divide it by the PRs shipped. At ~$30/PR, it reframes the entire ROI conversation: the spend isn't the problem, spend with no return is.

This episode comes from a live panel Sam moderated at VentureBeat Transform, featuring Emilie Schario (Kilo Code, acquired by Anaconda the morning of the panel), Amol Jain (Replit), and Jared Go (Symbiotic). The conversation cuts straight to what agentic coding actually looks like in production — not in demos.

On architecture: Kilo routes across 500+ models and processes 10 trillion tokens/month, using expensive frontier models for planning and cheaper open-weight models for execution. Replit built an internal fleet of agents running in isolated cloud VMs — connected to GCP, Datadog, and Slack — that autonomously handled a gnarly multi-day bug and produced a merged PR with zero human intervention. Symbiotic runs tiered monthly token budgets ($150 / $450 / exception) managed directly by engineering managers, not finance.

On org design: code review — not code generation — is now the bottleneck. Replit risk-scores every PR; low-risk ones auto-merge. The panel is aligned that "what does code review mean in 2025" is the defining question for the next 6–12 months.

On the 6-month outlook: open-weight models take majority share of coding tasks, frontier models reserved for architecture decisions; 24–48 hour unattended agent work loops become standard; and model sovereignty (geographic data routing, on-prem isolated environments) becomes a hard enterprise requirement.

---

🎙️ GUESTS: Emilie Schario | Head of Engineering, Kilo Code (acquired by Anaconda) | Amol Jain | VP Engineering, Replit | Jared Go | Co-founder, Symbiotic

🎙️ HOST: Sam Witteveen | VentureBeat

---

00:00 Intro & Cost Per PR framing

00:40 Panel intro: Kilo Code, Symbiotic, Replit

02:20 Do engineers still read code? (Instacart CTO's 97% stat)

03:00 How Replit uses risk-scored PRs to auto-merge

04:20 Physical AI and safety gates for code review (Symbiotic)

05:00 Kilo Code: 500+ models, 10T tokens/month, Anaconda acquisition

07:20 Multi-model routing: frontier for planning, open-weight for execution

09:40 Symbiotic's stack: Cursor + homegrown SymCloud agentic harness

11:20 Replit's internal self-driving software engineering tool

12:20 MCP servers, skills repositories, and the 6-hour autonomous bug fix

13:20 Build vs. buy: where the real cost is maintenance, not construction

15:40 Replit replaces a seven-figure SaaS tool built entirely in-house

17:40 Token maxing: how teams burned their annual AI budget by July

18:20 Cost per PR as a ROI metric for AI coding spend

21:00 Symbiotic's tiered monthly token budget system for 900 engineers

23:20 Replit's 3x engineering productivity — and the GPT-5.5 Pro Max bill that forced a reckoning

24:20 Audience Q&A: model routing as a product layer for ISVs

26:20 Audience Q&A: PR review bottleneck, code deletion, and what review looks like next

29:40 6-month predictions: open-weight dominance, 24–48hr agent work loops, model sovereignty

---

Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat

Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239

Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4

Website: https://venturebeat.com

LinkedIn: https://www.linkedin.com/company/venturebeat

Newsletter: https://venturebeat.com/newsletters

#EnterpriseAI #AgenticCoding #AIAgents #LLMInfrastructure #AIDeployment

---

Learn more about your ad choices. Visit megaphone.fm/adchoices

66% of enterprise AI teams are actively engineering toward zero human oversight — but only 4.5% fully trust their automated evals. Half have already shipped an agent that passed evals and still failed a real customer.

VentureBeat's latest research exposes a dangerous gap between deployment ambition and evaluation maturity. The most-used eval platform is OpenAI's native tooling. The second most common answer: no dedicated eval platform at all. Meanwhile, 51% of enterprises are relying on a hyperscaler's built-in guardrails as their primary AI security layer — not a deliberate security architecture.

Fable's 19-day market absence changed the conversation. Enterprises that had built production workflows on frontier models got a wake-up call about single-vendor dependency. The result: GLM 5.2 is gaining serious developer traction as an open alternative competitive with Opus 4.8, and Tencent's new Hai 3 — less than half the size — is drawing attention for agentic workloads that don't require cutting-edge coding performance. Sam and Matt break down why companies like Shopify, LinkedIn, and Pinterest were already on open models, and why others are now catching up fast.

On infrastructure: neo-clouds are currently used by just 2% of respondents, but 45% name them as their top planned investment area over the next 12 months. The market hasn't calcified. Matt and Sam preview what they expect to hear at VentureBeat Transform from engineering leaders at Intuit, Brex, Instacart, and Mastercard — including how Intuit bootstrapped and pivoted its agent architecture mid-build, and why agent conversations are generating higher-quality behavioral data than traditional user research ever did.

🎙️ HOSTS: Matt Marshall · Sam Witteveen | VentureBeat

__

If you enjoy these conversations, you need to be in Menlo Park this July.

VB Transform 2026 is VentureBeat's flagship enterprise AI event, built entirely around one question: How do you orchestrate AI autonomy at scale? July 14–15, Hotel Nia. Real projects, proprietary research, no fluff.

50% off for listeners with code BEYONDTHEPILOT: https://bit.ly/4fK4F6z

—

Chapters:

00:00 Research data drop: eval gaps, zero-HITL targets, and hyperscaler security dependency

01:00 Chinese models breaking into enterprise — developer fad or structural shift?

02:00 GLM 5.2 and Tencent Hai 3: what they do well and where they fall short

03:20 How Shopify, LinkedIn, and Pinterest are already running open models in production

05:40 Fable's 19-day outage: why 19 days changed enterprise AI vendor strategy

07:00 Intuit's agent build: bootstrapping, mid-build pivots, and real-time behavioral data

08:40 Neo-cloud infrastructure: 2% current usage, 45% planned investment next 12 months

09:20 Is the enterprise AI market still wide open? Anthropic vs. OpenAI for enterprise lead

11:20 The routing question: when do you send workloads to smarter vs. cheaper models?

12:20 Transform preview: what to expect from Amazon, Databricks, Brex, Instacart, Mastercard

---

Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat

Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239

Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4

Website: https://venturebeat.com

LinkedIn: https://www.linkedin.com/company/venturebeat

Newsletter: https://venturebeat.com/newsletters

#EnterpriseAI #AIAgents #LLM #MLOps #AgenticAI

—

Learn more about your ad choices. Visit megaphone.fm/adchoices

Shopify's distillation pipeline cuts production AI costs by up to 30x — and in some cases, the smaller model outperforms the frontier model on the narrow task. That's not a trade-off. That's a win on accuracy, latency, and cost simultaneously.

Farhan Thawar, VP & Head of Engineering at Shopify, runs AI across one of the largest commerce platforms on earth. In this episode, he breaks down the exact infrastructure decisions Shopify made to avoid being locked into any single model provider — and why 29% of enterprise AI projects die from token costs, not model failure.

Shopify built an internal LLM proxy that routes tokens across every major provider, enabling automatic failover when any one goes down. On top of that, their Universal Distillation Platform (UDP) lets any R&D team distill a frontier model (Opus 4, GPT-5+) down to a fine-tuned open source model (Qwen and others) for a specific subtask — in roughly a day, with evals baked in. Results range from 2x to 30x cheaper, faster, and more accurate than calling the frontier API for everything. Shopify currently runs roughly half a dozen of these distilled models in production, with more being added.

Farhan also details River, Shopify's internal agentic substrate — a public-only Slack agent that queries their data warehouse, reads their PM system, and improves its own answers when engineers jump in to correct it. Plus: how Shopify governs AI-generated code at scale, why they killed their token leaderboard, how UCP is positioning Shopify's catalog for agentic commerce, and what a two-to-three year horizon looks like when agents start holding spending budgets and buying autonomously.

🎙️ GUEST: Farhan Thawar | VP & Head of Engineering, Shopify

🎙️ HOST: Sam Witteveen | VentureBeat

__

If you enjoy these conversations, you need to be in Menlo Park this July.

VB Transform 2026 is VentureBeat's flagship enterprise AI event, built entirely around one question: How do you orchestrate AI autonomy at scale? July 14–15, Hotel Nia. Real projects, proprietary research, no fluff.

50% off for listeners with code BEYONDTHEPILOT: https://bit.ly/4fK4F6z

—

**CHAPTERS**

00:00 Intro — Infrastructure-first philosophy at Shopify

02:00 Episode overview: token economy, distillation, and the cost crisis killing AI projects

03:00 Toby's "AI reflexivity" mandate and what it actually means for engineers

05:20 Ecosystem strategy: when to build vs. leave room for third-party developers

06:20 Developer tooling stack: GitHub Copilot (2021), Claude Code, Cursor, Codex, and Shopify's own River

08:00 LLM proxy architecture: bulk token purchasing, multi-provider failover, and usage reporting

09:00 Model agnosticism: why Shopify lets engineers choose their own harness

10:00 AI adoption beyond R&D — finance, HR, sales, and the Qwik internal deployment platform

11:20 Code governance: who owns AI-generated code going to production

13:00 Token maxing, leaderboards, and the shift from AI reflexivity to AI leverage

14:20 Circuit breakers: how Shopify catches runaway token spend without hard limits

15:20 LLM proxy deep dive: uptime, insights, and cross-team learning from usage data

16:40 River: Shopify's agentic substrate, public-channel-only design, and emergent HITL behavior

18:40 Model distillation explained: teacher/student models, narrow tasks, and the trade-offs

21:00 Universal Distillation Platform (UDP): how any team submits a distillation job in ~one day

22:20 Tangle: open source pipeline visualization for distillation workflows

22:40 Who uses UDP today, and Farhan's vision for auto-selecting the distillation target model

24:00 Evals in practice: golden datasets, Toloka for data generation, threshold-based deployment

26:20 Sim Gym: simulating A/B tests for small merchants without enough traffic

27:40 Pulse: async AI insights on store performance and conversion

28:20 GPU infrastructure trade-offs: when running your own inference makes sense at scale

29:20 Frontier vs. distilled model split in production — and why dev tokens stay frontier

30:40 Should Shopify train its own coding model? Why Farhan says not yet

31:40 UCP protocol, agentic commerce, and how Shopify's catalog surfaces in every LLM

35:00 Early signals: agentic commerce growth rate and the shift away from SEO

36:00 What developers and entrepreneurs should build for now given multi-channel uncertainty

37:20 River as a hive-mind agent — and what "truly agentic" actually means

38:40 Two-to-three year forecast: agents with spending budgets, autonomous purchasing, proactive outreach

40:00 Project Glasswing, model access loss (Fable), and the case for multi-provider architecture

42:00 Why every company should have a backup plan — and how Shopify built theirs years ago

43:20 Wrap-up

---

Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat

Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239

Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4

Website: https://venturebeat.com

LinkedIn: https://www.linkedin.com/company/venturebeat

Newsletter: https://venturebeat.com/newsletters

#EnterpriseAI #AIAgents #LLM #MLOps #AgenticAI

—

Learn more about your ad choices. Visit megaphone.fm/adchoices

If you enjoy these conversations, you need to be in Menlo Park this July.

VB Transform 2026 is VentureBeat's flagship enterprise AI event, built entirely around one question: How do you orchestrate AI autonomy at scale? July 14–15, Hotel Nia. Real projects, proprietary research, no fluff.

50% off for listeners with code BEYONDTHEPILOT: https://bit.ly/4fK4F6z

_

He negotiated seat-based (unlimited) licenses before token costs exploded — and projects only a 20–30% spend increase when that changes. MassMutual's CIO rebuilt a COBOL mainframe app into a working web prototype in 7 days — work that used to take a 15-person SI team 90 days. That's not a pilot. That's a new build-vs-buy equation.

Sears Merritt, CIO at MassMutual, runs AI inside one of America's most regulated legacy environments. In this conversation, he breaks down the architecture decisions, cost structures, and security posture behind real, production deployments — not roadmaps.

On infrastructure: MassMutual routes all agentic tool calls through centralized API gateways with identity and access controls, using Amazon Bedrock as a proxy layer. That multi-harness design preserves model optionality while enforcing FinOps discipline. On model selection: a trust score rubric drives every model decision, balancing cost against user experience. In their IT contact center, that rubric led them to choose the more expensive model after users said the quality gap was worth two extra seconds of inference time.

Productivity results are concrete: 30% boost in developer output across the SDLC, call resolution time dropping from 10 minutes to under 1 minute for specific call types, cost per interaction from dollars to cents. On the security side, MassMutual is embedding AI into its SDLC for vulnerability scanning and compressing cyber response cycles from days to hours — building agentic tier-one and tier-two capabilities to match the accelerated threat landscape that frontier models like Mythos have exposed.

🎙️ GUEST: Sears Merritt | Head of Enterprise Technology & Experience, MassMutual

🎙️ HOSTS: Matt Marshall | VentureBeat, Sam Witteveen | VentureBeat

—

If you enjoy these conversations, you need to be in Menlo Park this July.

VB Transform 2026 is VentureBeat's flagship enterprise AI event, built entirely around one question: How do you orchestrate AI autonomy at scale? July 14–15, Hotel Nia. Real projects, proprietary research, no fluff.

50% off for listeners with code BEYONDTHEPILOT: https://bit.ly/4fK4F6z

—

00:00 Intro & COBOL Modernization Preview

00:01:15 Guest Introduction: Sears Merritt, MassMutual CIO

00:02:00 Multi-Vendor Strategy & Avoiding Lock-In

00:02:30 How MassMutual Evaluates AI Tools (Cost vs. Experience Rubric)

00:03:15 12-Month Contracts and Switching Optionality

00:03:30 AI Standards Cycle: MCP, A2A, and the Early Internet Analogy

00:05:15 Measuring Developer Productivity: 30% SDLC Boost

00:06:15 Contact Center Results: 10 Minutes to 1 Minute, Dollars to Cents

00:07:00 Managing Token Cost Explosion

00:07:30 Seat-Based vs. Consumption Licensing Decision

00:08:45 Token Maxing While the All-You-Can-Eat Window Is Open

00:09:45 Building FinOps Infrastructure for Model Routing and Optimization

00:11:15 Outcome-First Model Selection: When to Pay for Opus vs. a Cheaper LLM

00:13:15 Trust Score Framework: How MassMutual Picks the Right Model

00:15:00 Sponsor: OutShift by Cisco

00:15:30 Claude, OpenAI Codex, and Multi-Harness Agentic Architecture

00:16:30 API Gateway Design: Identity, Access, and FinOps Controls

00:17:30 What the Usage Analytics Revealed (And What Merritt Was Afraid to Find)

00:18:15 Projected Token Cost Increase: 20–30% Off Unlimited Plan

00:19:45 Power Law Usage: Top 10% Consuming 80% of Tokens

00:20:45 COBOL Mainframe Modernization: The 7-Day Prototype Workflow

00:22:30 The Full AI-Assisted COBOL Migration Playbook

00:24:00 Implications for IBM and Mainframe-as-a-Service Providers

00:25:15 Open Source Models, DeepSeek, and the Cost Efficiency Question

00:27:45 Chinese Models in a Regulated Environment: Evaluation Criteria

00:29:30 Agentic Security: Identity Management and the Evolving Threat Landscape

00:30:00 How Frontier Models Changed the Threat Velocity (Not the Threat Types)

00:31:15 Fighting AI With AI: Agentic Tier-1 and Tier-2 Cyber Capabilities

00:31:30 Project Glasswing and the CISO Community Response

00:32:30 Embedding AI Into the SDLC for Security Scanning

00:34:00 Closing: When Will Agentic Standards Consolidate? Advice for Builders

---

Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat

Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239

Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4

Website: https://venturebeat.com

LinkedIn: https://www.linkedin.com/company/venturebeat

Newsletter: https://venturebeat.com/newsletters

#EnterpriseAI #AIAgents #LLMInfrastructure #AIDeployment #MLOps

Learn more about your ad choices. Visit megaphone.fm/adchoices

Pinterest's open-source AI stack costs 90% less than frontier models — and their custom-trained recommender outperforms off-the-shelf alternatives by 30% in accuracy. Pinterest CTO Matt Madrigal breaks down exactly how they did it, and what enterprise AI teams can actually replicate.

Madrigal walks through the full architecture behind Navigator 1, Pinterest's conversational shopping assistant built on Qwen 3 VL — and the specific decision to rip out its native vision encoder and replace it with PinCLIP, Pinterest's proprietary multimodal embedding layer. That swap alone closes a 20x inference latency gap and makes the economics work at 620 million monthly active users. This is the clearest public explanation yet of how a scaled platform operationalizes the "core vs. context" principle for model selection: open-source and custom-built where it touches the user, frontier models where speed-to-prototype matters more than cost.

The conversation also covers the Taste Graph — Pinterest's knowledge graph across hundreds of billions of pins and 15 billion boards — and how post-training on that proprietary data lets a smaller, fit-for-purpose model beat a larger frontier model on production metrics. Madrigal details their eval framework: gold set benchmarks, product-level evals tied to engagement and merchant click outcomes, and a structured A/B test pipeline that runs from engineer PRs through to live user signal.

On the organizational side: how Pinterest manages a "default yes" multi-IDE policy (Cursor, Windsurf, Claude Code, Codex) without collapsing security posture, how they segment sandbox environments between ML engineers with Taste Graph access and general application developers, and why Madrigal measures AI coding ROI in token usage and experimentation velocity — not lines of code.

🎙️ GUEST: Matt Madrigal | CTO, Pinterest

🎙️ HOSTS: Matt Marshall | VentureBeat, Sam Witteveen | VentureBeat

00:00 Show Intro and Guest

01:17 Open Source Cost Breakdown

02:20 Pinterest Multimodal Roots

02:37 PinClip and Embeddings

05:46 Core vs Context Models

07:43 Navigator 1 Assistant Stack

11:52 Benchmarking and Evals

13:29 Accuracy from Proprietary Data

17:16 Taste Graph Explained

18:29 Taste Graph in Training

22:22 Fighting AI Slop

25:16 Developer Tools and Velocity

27:57 Tool Choice and Governance

28:56 Security Sandboxes and CICD

30:57 Wrap Up

If you enjoy these conversations, you need to be in Menlo Park this July.

VB Transform 2026 is VentureBeat's flagship enterprise AI event, built entirely around one question: How do you orchestrate AI autonomy at scale? July 14–15, Hotel Nia. Real projects, proprietary research, no fluff.

50% off for listeners with code BEYONDTHEPILOT: https://bit.ly/4fK4F6z

Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat

Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239

Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4

Website: https://venturebeat.com

LinkedIn: https://www.linkedin.com/company/venturebeat

Newsletter: https://venturebeat.com/newsletters

#EnterpriseAI #OpenSourceAI #AIInfrastructure #LLM #MachineLearning

Learn more about your ad choices. Visit megaphone.fm/adchoices

Enterprise GPU hoarding is over. LinkedIn CTO Erran Berger and VentureBeat analyst Rob Strechay break down what comes next — and the infrastructure math most enterprises are only now being forced to confront.

VentureBeat's Q1 research shows GPU availability anxiety dropped from 20.8% to 15.4% among enterprise teams, while cost-per-inference and TCO concerns jumped from 34% to 41% — a number that's still climbing. The hoarding phase is giving way to an audit phase, and the companies that didn't build the instrumentation to understand their workloads are now paying for it.

Erran Berger explains how LinkedIn runs one of the few remaining at-scale applied ML shops outside the hyperscalers — owning the full stack from bare metal GPU clusters to member-facing products. That means LinkedIn engineers can optimize custom CUDA kernels, compress embeddings, prune models for throughput, and adapt networking and storage per workload — trade-offs that are simply unavailable on public cloud instance menus. The result: a rigorous ROI framework that evaluates not just current traffic costs, but the traffic shape agents will drive in 2–3 years.

On the market side, 72% of enterprises admit they lack sufficient control over their AI infrastructure. Open-source inference tools like vLLM and LLMD are seeing rapid adoption, while 17% of organizations have moved to full-stack ownership. Hyperscalers report 60–80% of workloads have already shifted from training to inference — and most enterprise teams are still figuring out how to staff and instrument for that reality.

🎙️ GUEST: Erran Berger | CTO, LinkedIn

🎙️ ANALYST: Rob Strechay | VentureBeat

🎙️ HOST: Matt Marshall | CEO, VentureBeat

---

00:00 Intro: The GPU Hoarding Hangover

00:10 Guest Introductions

02:00 VentureBeat Q1 Data: GPU Panic Fades, TCO Concerns Rise

03:00 LinkedIn's Early Shift to Inference ROI Discipline

04:00 Budget Moving Into Inference Optimization and Control

07:00 LinkedIn's Full-Stack Advantage: Kernels, Pruning, Embedding Compression

08:00 Private AI and Sovereign Stacks: What the Q1 Data Shows

09:00 Open Source Inference Tooling: vLLM, LLMD, RDMA

10:00 Data Sovereignty at LinkedIn Scale: Member Data and Board-Level ROI Framing

12:00 Why Instrumentation Beats GPU Hoarding

13:00 Planning for Ambient Agent Traffic — Not Just Today's Workloads

14:00 Closing Advice for the Enterprise CTO Staring at 5% GPU Utilization

---

Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat

Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239

Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4

Website: https://venturebeat.com

LinkedIn: https://www.linkedin.com/company/venturebeat

Newsletter: https://venturebeat.com/newsletters

#EnterpriseAI #AIInfrastructure #MLOps #InferenceOptimization #GenerativeAI

---

Learn more about your ad choices. Visit megaphone.fm/adchoices

The CEO who built one of the most-starred RAG frameworks on GitHub (47,000 stars) just publicly declared that frameworks like his are becoming obsolete — and then pivoted his entire company around that conclusion.

Jerry Liu, CEO and co-founder of LlamaIndex, joins Matt Marshall and Sam Witteveen to explain exactly what broke in the AI stack, why 95% of his team's code is now AI-generated, and where the real defensibility in enterprise AI infrastructure actually lives in 2026.

The conversation covers the specific architectural shift that made RAG orchestration frameworks less central: agent reasoning has improved to the point where dumb tools plus smart agents outperform sophisticated retrieval pipelines, coding agents have collapsed the cost of custom integrations, and model providers like Anthropic are consolidating the harness layer around MCP, sandboxes, and session state. Jerry walks through Anthropic's managed agent diagram as a real architectural reference point and explains why engineering leaders should prioritize modular interfaces over implementation investment — because parts of your current stack will need to be thrown away in months, not years.

On SaaS survival, Jerry argues the companies that retain value are those becoming systems of record — and that the real opportunity is building AI agents that automate labor on top of their platforms, not defending UI/UX that agents are now bypassing. On LlamaIndex's own bet: document understanding — parsing PDFs, tables, charts, and forms at higher accuracy and lower cost than frontier models — is the context layer every agent stack needs regardless of which model wins the next benchmark cycle. LlamaParse and the newly released open-source ParseBench (April 13) are the commercial expression of that thesis.

If you're evaluating your AI stack architecture, deciding how much to build vs. buy, or trying to understand where horizontal tooling still has a moat, this episode is the conversation.

🎙️ GUEST: Jerry Liu | CEO & Co-Founder, LlamaIndex

🎙️ HOSTS: Matt Marshall | VentureBeat, Sam Witteveen | VentureBeat

---

**CHAPTERS**

00:00 Intro — LlamaIndex's origin and RAG framework origins

02:00 How LlamaIndex started: GPT-3, 4K context windows, and GPT Index

04:00 Why AI frameworks are becoming less useful in the agentic era

07:00 What changed in the stack: agent reasoning, coding agents, and RAG's evolution

09:00 How Anthropic's managed agent diagram reframes enterprise architecture

13:00 The lock-in question: managed agents, session state, and stack modularity

16:00 Should you build horizontal tooling? Why Jerry says probably not

18:00 Open vs. closed: the Apple/Android analogy applied to frontier labs

21:00 The abstraction level is rising — English is the new programming language

24:00 SaaS market cap destruction: who survives agents eating software

28:00 The "full stack builder" emergence and the future of SaaS seats

31:00 Buy vs. build for agents: the AI recruiter thought experiment

33:00 LlamaIndex's pivot: document understanding as defensible infrastructure

36:00 Why frontier models won't commoditize specialized document parsing

38:00 LlamaParse deep dive: zero-shot accuracy, tables, charts, handwriting

41:00 LightParse, ParseBench, and designing for agent consumers

44:00 Wrap-up and where to follow LlamaIndex

---

Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat

Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239

Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4

Website: https://venturebeat.com

LinkedIn: https://www.linkedin.com/company/venturebeat

Newsletter: https://venturebeat.com/newsletters

#EnterpriseAI #AIAgents #LLMInfrastructure #RAG #AIArchitecture

---

Learn more about your ad choices. Visit megaphone.fm/adchoices

Cisco's OutShift deployed a multi-agent network configuration system that raised error detection from 10–15% to 100% and cut full change validation from 2–3 weeks to 6–7 minutes. The reason it worked — and why most enterprise multi-agent deployments still fail — comes down to a single gap nobody is talking about: agents can connect, but they cannot think together.

Vijoy Pandey, SVP and General Manager of OutShift by Cisco, joins Matt and Sam to explain why A2A, MCP, and existing agent protocols solve connectivity but leave out an entire layer: shared cognition. OutShift's research identifies this as a missing "Layer 9" — a semantic and cognitive communication stack above today's syntactic protocols — and they're already building it.

The conversation covers the four pillars of enterprise-grade multi-agent infrastructure (discovery, identity/access, communication, observability), why standard IAM models break when agents enter the picture, and how OutShift extended OpenTelemetry with Microsoft to cover multi-agent evaluation. Vijoy introduces three new cognition-state protocols — SSTP (Semantic State Transfer), LSTP (Latent Space Transfer), and CSTP (Compressed State Transfer) — and explains the staged rollout path for each, including a published MIT collaboration called the Ripple Effect Protocol.

The healthcare scheduling case study is particularly instructive: three independent third-party agents — insurance, diagnostics, scheduling — each with competing optimization functions and siloed context, and zero shared intent. That's the real multi-vendor, multi-org enterprise problem. Vijoy explains what an orchestrator can't fix, and what a cognitive fabric layer would.

🎙️ GUEST: Vijoy Pandey | SVP & General Manager, OutShift by Cisco

🎙️ HOSTS: Matt Marshall | VentureBeat, Sam Witteveen | VentureBeat

---

**CHAPTERS**

00:00 Intro & Cold Open: Agents Connect But Can't Think Together

00:03 Welcome & Guest Introduction: Vijoy Pandey, OutShift by Cisco

00:04 Do Agents Work Outside Coding & Customer Support? Challenging Amjad Masad's Diagnosis

00:05 What's Wrong With A2A and MCP? The Four Pillars of AGNTCY

00:08 Identity & Access Management for Agents: Why IAM Breaks and What TBAC Fixes

00:12 The Network Digital Twin: How OutShift Achieved 100% Error Detection in Production

00:13 From 2–3 Weeks to 6–7 Minutes: Real Results From Deployed Multi-Agent Networking

00:15 Agents Can Connect But Can't Think Together: The Core Thesis

00:20 The Cognitive Revolution Analogy: Shared Intent, Shared Context, Collective Innovation

00:25 The Healthcare Scheduling Case Study: Three Competing Agents, Zero Shared Intent

00:31 Why Orchestrators Fail in Multi-Vendor, Multi-Org Environments

00:36 Introducing Layer 9: SSTP, LSTP, and CSTP — The Cognition-State Protocol Stack

00:41 What OutShift Is Building Now: Protocols, Fabric, and Cognition Engines

00:44 MIT Collaboration: The Ripple Effect Protocol and Phase One Rollout

00:46 Cisco's 40-Year Networking Playbook Applied to the Internet of Cognition

00:49 Closing: Where to Find the Research, AGNTCY, and OpenClaw Integration

---

Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat

Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239

Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4

Website: https://venturebeat.com

LinkedIn: https://www.linkedin.com/company/venturebeat

Newsletter: https://venturebeat.com/newsletters

#EnterpriseAI #AIAgents #MultiAgentSystems #AIInfrastructure #LLM

—

“Scaling Out Superintelligence”  Vijoy Pandey, January 2026. The technical whitepaper detailing the Internet of Cognition architecture, three-layer stack, and cognition state protocols. 

Internet of Cognition Interactive Demo Clickable walkthrough showing per-agent activity, intent, context, and collective reasoning across a multi-agent SRE system. 

“A Layered Protocol Architecture for the Internet of Agents”  Fleming, Muscariello, Pandey, Kompella. The OSI Layer 8/9 extension. 

AGNTCY Open source multi-agent infrastructure under Linux Foundation governance. Covers discovery, identity, communication, observability. 

Formative members: Cisco, Dell Technologies, Google Cloud, Oracle, Red Hat. 

Learn more about your ad choices. Visit megaphone.fm/adchoices