As enterprises deploy AI agents into production, a new operational challenge emerges: how do you monitor and maintain systems that don't fail with error codes, but instead drift subtly away from expected performance? In this episode, the AI, Actually crew tackles the emerging discipline of AgentOps—the practice of keeping AI agents performing at peak business value over time.

The discussion cuts through the hype around "self-learning" and "automated" AI to reveal the hard truth: agentic systems require continuous human oversight, just like human employees do. From probabilistic model behavior to reasoning model complexity, the team explores why traditional IT monitoring approaches fall short and why businesses need to rethink who owns these digital workers.

Topics covered:

• Why AgentOps is fundamentally different from traditional DevOps and LiveOps
• The three levels of agent complexity and three types of drift that can derail performance
• Why traditional IT support models don't work for goal-driven agentic systems
• The organizational challenge of bringing together business knowledge, AI expertise, and technical skills
• Why there's no "blue screen of death" for agent failures—and what that means for monitoring

Follow the Gang:

• Nicole Kosky, Senior Director of Services, AnswerRocket | https://www.linkedin.com/in/nicole-kosky-5b9a3b6/
• Joey Gaspierik, Director of Enterprise Sales, AnswerRocket | https://www.linkedin.com/in/joey-gaspierik-4a613642/
• Stew Chisam, Operating Partner, StellarIQ - https://www.linkedin.com/in/stewart-chisam-7242543/ 
• Jim Johnson, Managing Partner, AnswerRocket | https://www.linkedin.com/in/jim-johnson-bb82451/

Chapters:

00:00     Introduction to AgentOps

02:38     Defining Agentic Operations

09:30     The Role of Human Oversight

11:01     Understanding Performance Degradation

17:27     The Complexity of Monitoring Agents

26:07     Organizational Challenges in AgentOps

31:00     The Future of Agentic Operations

35:26     What's An Agent?

Hashtags: #AIImplementation #EnterpriseAIAdoption #OrganizationalChange #AILiteracy #GeneralistEngineers #LastMileProblem #VibeCoding #AIROI #RevenueGeneration #OpenAIWhitePaper

Keywords: Agent Ops, AI agents, enterprise AI, LLM monitoring, model drift, probabilistic systems, agentic AI, AI operations, AI governance, AI deployment, DevOps, LiveOps, reasoning models, AI scalability, digital workers

The old IT playbook is officially dead. Quarterly release cycles, endless approval committees, and throwing requirements over the wall? None of that works when AI models evolve every few weeks. In this episode, Pete Reilly sits down with Jim Johnson, Alon Goren, and Shanti Greene to unpack OpenAI's new white paper, "From Experiments to Deployments: A Practical Path to Scaling AI," and share what they're seeing on the ground with enterprise clients.

This isn't theory—it's a frontline report from teams who are helping Fortune 500 companies navigate the messy reality of AI adoption. The conversation tackles the organizational upheaval required to move fast, the "last mile problem" that kills so many AI projects, and why the future belongs to curious generalists who can bridge business and technology. They also explore how AI is shifting ROI conversations from cost savings to revenue generation, and why vibe-coding a working prototype in hours is now entirely possible (with some important caveats about technical debt).

Topics covered:

• Why IT and business teams must close the collaboration gap to succeed with AI
• The "last mile problem": getting from 80% complete to actually useful
• How generalists with medium depth are becoming more valuable than deep specialists
• Building an AI-literate workforce through curiosity and continuous learning
• Shifting from cost-savings ROI to revenue-generating AI products
• The reality of vibe-coding: prototypes in hours, but engineering rigor still matters

Follow the Gang:

• Pete Reilly, COO, AnswerRocket - https://www.linkedin.com/in/petereilly 
• Shanti Greene, Head of Data Science and AI Innovation, AnswerRocket - https://www.linkedin.com/in/shantigreene/ 
• Alon Goren, Founder, AnswerRocket | https://www.linkedin.com/in/alon-goren-87889681/
• Jim Johnson, Managing Partner, AnswerRocket | https://www.linkedin.com/in/jim-johnson-bb82451/

Chapters:

00:00     Introduction to AI Deployment Challenges

02:52     The Shift from IT to Business Collaboration

05:41     The Role of Generalists in AI Integration

08:42     Redefining AI Projects and Business Goals

11:35     Accelerating Development Timelines with AI

14:47     Building an AI Literate Workforce

17:45     The Changing Landscape of ROI in AI

20:58     Final Thoughts on Embracing AI

#AIImplementation #EnterpriseAIAdoption #OrganizationalChange #AILiteracy #GeneralistEngineers #LastMileProblem #VibeCoding #AIROI #RevenueGeneration #OpenAIWhitePaper

Keywords: AI implementation, enterprise AI adoption, organizational change, AI literacy, generalist engineers, last mile problem, vibe coding, AI ROI, revenue generation, OpenAI white paper

The race for AI dominance just shifted. Google's Gemini 3 launch was a coordinated ecosystem play that could reshape how enterprises think about AI infrastructure. In this episode, Pete Reilly sits down with Andy Sweet, Shanti Greene, and Stew Chisam to dissect what Gemini 3 really means for enterprise adoption, where the technology is genuinely improving, and where marketing hype obscures practical limitations.

The conversation moves beyond surface-level benchmarks to tackle the uncomfortable reality facing IT leaders: foundation models are converging on performance, but the real competitive advantage lies in how you architect solutions on top of them. The team explores Google's commanding lead in multimodal capabilities, the strategic implications of vendor ecosystems, and why enterprises betting everything on a single provider might be making a costly mistake. Then they close with their boldest predictions for 2026, from content exhaustion and the death of infographics to agents working autonomously longer than your employees.

What You'll Learn:

• How Gemini 3's pre-training approach signals continued model improvements and what that means for the scaling law debate
• Why Google's multimodal dominance (backed by YouTube, Google Photos, and Drive) creates a moat that competitors can't easily replicate
• The critical difference between general intelligence benchmarks and enterprise intelligence that understands your business context
• Why the "stateless" problem keeps plaguing AI solutions and how memory scaffolding becomes essential for business applications
• Predictions for 2026: content exhaustion, shadow IT proliferation, and the moment enterprises realize there's no easy button

Follow the Gang:

• Shanti Greene, Head of Data Science and AI Innovation, AnswerRocket - https://www.linkedin.com/in/shantigreene/ 
• Pete Reilly, COO, AnswerRocket - https://www.linkedin.com/in/petereilly 
• Andy Sweet, VP Enterprise AI Solutions, AnswerRocket - https://www.linkedin.com/in/andrewdsweet/ 
• Stew Chisam, Operating Partner, StellarIQ - https://www.linkedin.com/in/stewart-chisam-7242543/ 

Chapters:
00:00     Introduction to Gemini 3 and Episode Overview
01:45     Gemini 3's Long-Term Planning Capabilities
04:17     Are LLMs Becoming Commoditized Primitives?
08:04     Model Specialization and Jagged Edges
11:15     Why Multi-Vendor Strategy Matters for Enterprises
13:29     OS/2 vs Windows: Best Doesn't Always Win
15:10     The Scaling Law Debate and Pre-Training Improvements
16:12     Enterprise Intelligence vs General AGI
22:41     How Enterprises Should Think About Gemini 3
26:44     Bold Predictions for 2026
28:08     Content Exhaustion and the Infographic Problem
33:06     Agents as Autonomous Team Members

#Gemini3 #EnterpriseAIArchitecture #MultimodalAI #AIAgents #VendorLockIn #SemanticLayer #ScalingLaws #PreTrainingCompute #EnterpriseIntelligence #AIPredictions2026

Keywords: Gemini 3, enterprise AI architecture, multimodal AI, AI agents, vendor lock-in, semantic layer, scaling laws, pre-training compute, enterprise intelligence, AI predictions 2026

Are most AI projects really failing, or are we just too early to judge? In this episode, we cut through the headlines to talk about what's actually happening with enterprise AI adoption. Our team tackles the disconnect between AI hype and business reality, exploring why the "easy button" mentality is holding companies back and what it actually takes to succeed.

We discuss why treating AI like magic instead of a project dooms initiatives from the start, how mid-market companies have a unique advantage to leapfrog their larger competitors, and why someone needs to supervise your digital employees. This is a practical conversation about bridging the gap between IT departments and business aspirations, with real talk about what works and what doesn't.

What You'll Learn:

• Why "AI failure" statistics miss the point about early adoption
• The three critical skills needed for successful AI implementation
• How to identify which use cases deserve an LLM versus deterministic code
• What "agentic operations" means and why it's not set-it-and-forget-it
• Why mid-market companies can leapfrog enterprise competitors right now
• How to bridge the dangerous gap between IT and business expectations

Follow the Gang:

Jim Johnson, Managing Partner, AnswerRocket | https://www.linkedin.com/in/jim-johnson-bb82451/

Nicole Kosky, Senior Director of Services, AnswerRocket | https://www.linkedin.com/in/nicole-kosky-5b9a3b6/
Joey Gaspierik, Director of Enterprise Sales, AnswerRocket | https://www.linkedin.com/in/joey-gaspierik-4a613642/
Shanti Greene, Head of Data Science and AI Innovation, AnswerRocket |

Chapters:
00:00     Introduction to AI Engagements
01:23     The Reality of AI Success and Failure
07:37     Deterministic vs Non-Deterministic Systems
11:12     Understanding Variation in AI Answers
13:35     Predictors of Success in AI Projects
16:41     Agent Operations and Ongoing Management
19:19     The Role of Senior Stakeholders in ROI
21:39     The Real Work To Do in Agent Operations
24:34     Putting Solutions Into Production
29:55     The Mid-Market AI Advantage
32:32     Closing: Recommendations for AI Success

#EnterpriseAIAdoption #AIProjectFailure #AgenticOperations #AIROI #GenerativeAIImplementation #LLMUseCases #MidMarketAIAdvantage #BusinessSponsorship #AISkillsGap #DeterministicVsNonDeterministicSystems

Keywords: enterprise AI adoption, AI project failure, agentic operations, AI ROI, generative AI implementation, LLM use cases, mid-market AI advantage, business sponsorship, AI skills gap, deterministic vs non-deterministic systems

Andrej Karpathy just dropped a reality check: 2025 isn't the "year of the agent,” it's the decade of the agent. But does that mean enterprises should hit pause on their AI initiatives? Not even close.

In this episode, the AI, Actually crew tackles the gap between AI hype and enterprise reality. Pete Reilly, Alon Goren, Mike Finley, and Jim Johnson break down why current LLMs aren't perfect, why that doesn't matter for ROI, and how smart companies are capturing value right now versus waiting ten years. From coding agents that ship features overnight to the organizational challenges of keeping AI solutions on the rails, this is your practical guide to navigating the AI landscape today.

Topics covered:

• Why Karpathy's agent timeline doesn't mean you should wait to start
• Where current AI models excel (and struggle) in enterprise environments
• The hidden engineering work between impressive demos and production systems
• Narrow use cases delivering ROI today: customer touchpoints, developer productivity, and sales pipelines
• The trade-offs between vendor lock-in and rapid implementation
• Why "it's just prompts" misses the entire point
• The three skill sets required to keep AI solutions running in production

Follow the Gang:

• Pete Reilly, COO, AnswerRocket | https://www.linkedin.com/in/petereilly/
• Alon Goren, Founder, AnswerRocket | https://www.linkedin.com/in/alon-goren-87889681/
• Mike Finley, CTO, AnswerRocket | https://www.linkedin.com/in/mikefinley/
• Jim Johnson, Managing Partner, AnswerRocket | https://www.linkedin.com/in/jim-johnson-bb82451/

Chapters: 

00:00     Introduction to AI Actually Podcast
02:22     The Decade of the Agent
05:45     Understanding AI Agents in the Enterprise
09:47     Navigating AI Use Cases
13:39     Building Modular AI Solutions
17:01     Identifying Low-Hanging Fruit for AI Implementation
20:29     The Importance of AI in Competitive Landscapes
24:36     Organizational Readiness for AI
28:16     Closing the Gap for Real Value in AI
32:06     Final Thoughts and Advice

#AIAgents #EnterpriseAIImplementation #AndrejKarpathy #LLMLimitations #AIROI #CodingAgents #AIVendorLockIn #AIDeployment #AIOrganizationalChange #EnterpriseAIStrategy

Keywords: AI agents, enterprise AI implementation, Andrej Karpathy, LLM limitations, AI ROI, coding agents, AI vendor lock-in, AI deployment, AI organizational change, enterprise AI strategy

OpenAI's Dev Day dropped some major announcements this month, but is AgentKit really revolutionary or just another "me too!" moment? The AI, Actually crew shares their reactions to the latest OpenAI releases, and digs into how to successfully implement AI agents in the real world.

In this episode, Jim Johnson steps in as host alongside Mike Finley, with special guests Nicole Kosky (who leads AnswerRocket's AI Business Transformation Practice) and Reilly Carrolll (Senior AI Solutions Consultant). Together, they tackle the practical, nitty-gritty challenges of bringing agents to life for enterprise clients—from gathering requirements that users don't know they have, to managing the surprising differences between what stakeholders say they need versus what they actually ask once they're hands-on with an agent.

Topics covered:

• OpenAI's Dev Day announcements and what they actually mean for enterprises
• Why successful agent implementations should follow the Software Development Lifecycle (SDLC)
• The critical role of context in agent performance
• Why user questions change dramatically from requirements phase to hands-on testing
• The emerging discipline of "agentic operations" and why it's non-negotiable
• Starting small: the power of quick wins over trying to boil the ocean

Follow the Gang:

Jim Johnson, AnswerRocket, Managing Partner - https://www.linkedin.com/in/jim-johnson-bb82451/
Mike Finley, AnswerRocket, CTO - https://www.linkedin.com/in/mikefinley/ 

Reilly Carroll, AnswerRocket, Senior AI Solutions Consultant - https://www.linkedin.com/in/reilly-carroll/ 

Nicole Kosky, AnswerRocket, Senior Director of Services - https://www.linkedin.com/in/nicole-kosky-5b9a3b6/ 

Chapters:

00:00      Introduction and Guest Introductions

01:30      OpenAI Dev Day Announcements

05:25      Understanding AI Agents

09:05      Practical Implementation of AI Agents

11:11      Challenges in Client Engagement

14:17      Agent Development and User Experience

16:55      The Challenge of Capturing Agent Requirements

19:52      The Importance of Agentic Operations

22:40      Navigating the Future of AI Agents

29:09      Final Thoughts and Advice

Keywords: AI agents, OpenAI Dev Day, agent development, enterprise AI implementation, agentic operations, software development lifecycle, AI business transformation, agent context, LLM applications, practical AI strategy

Tired of AI agents that forget context mid-conversation or drift subtly off course in production? You're not alone. In this episode, the AI, Actually crew unpacks six critical engineering principles for building reliable AI agents—principles that separate proof-of-concepts from production-ready systems.

Pete, Mike, Andy, and Stew break down insights from AI expert Nate B. Jones, translating technical concepts into business-focused guidance. They explore why AI memory isn't just about storage, how to bound uncertainty without killing creativity, and why monitoring AI systems requires a completely different approach than traditional software.

This episode covers:

• Why stateful intelligence and memory management are fundamental to useful AI interactions
• How to engineer controls that bound uncertainty without over-constraining your models
• The shift from binary failures to subtle quality drift in AI systems
• Capability-based routing: matching the right model to the right job
• Post-production monitoring strategies that catch problems before your users do
• Continuous validation techniques for multi-turn agent conversations

This episode of AI, Actually centers around a video by @nate.b.jones about the 6 principles of AI Agents. That video can be watched in its entirety here: I've Built Over 100 AI Agents: Only 1% of Builders Know These 6 Principles

Follow the Gang:

• Mike Finley, CTO, AnswerRocket - https://www.linkedin.com/in/mikefinley/ 
• Pete Reilly, COO, AnswerRocket - https://www.linkedin.com/in/petereilly 
• Andy Sweet, VP Enterprise AI Solutions, AnswerRocket - https://www.linkedin.com/in/andrewdsweet/ 
• Stew Chisam, Operating Partner, StellarIQ - https://www.linkedin.com/in/stewart-chisam-7242543/ 

Chapters: 

00:00    Introduction to AI Agents and Engineering Principles

01:34     Introducing Nate B. Jones' AI Engineering Principles

03:03    Stateful Intelligence

10:16     Bounded Uncertainty

19:55     Intelligent Failure Detection

20:51     Evaluating LLM Responses

22:16     Monitoring Quality and Performance

23:53    Active Maintenance of LLM Systems

26:18     Understanding Subtle Failures

26:55    Capability-Based Routing

30:22    Aligning Models with Business Processes

33:41     Nuanced Health State Monitoring

37:36     Continuous Input Validation

41:36     Closing Thoughts

Keywords: AI agents, agentic AI, AI engineering, AI memory, stateful intelligence, AI monitoring, capability-based routing, AI evaluation, production AI, enterprise AI, AI agent development, LLM engineering, AI testing, AI agent failures, AI system monitoring

Sequoia says AI is a $10 trillion opportunity. But how do you actually capture it? In this episode, the AI, Actually crew tackles the gap between AI's promise and its practical deployment in the enterprise. From bold predictions about agent automation to Palantir's forward deployed engineer model, we explore what it really takes to move beyond ChatGPT licenses to actual business transformation.

The discussion gets real about the current state of AI agents—including why they'd rather fake your data than admit they're stuck—and examines the critical role of specialization in making AI work for specific business processes. Whether you're trying to onboard suppliers across multiple systems or wondering why your coding agent just hardcoded the test answers, this episode provides the unfiltered truth about where enterprise AI stands today.

Key topics covered:

• Sequoia's $10 trillion services automation thesis and historical parallels to the steam engine
• The forward deployed engineer model and why proximity to the problem matters
• Current state of AI agents: what's working, what's broken, and why they love mocking up data
• The critical importance of specialization vs. general-purpose AI
• Real enterprise integration challenges beyond Google Drive
• Replit's Agent 3 and the evolution of autonomous coding tools

Follow the Gang:

• Alon Goren, Founder & CEO, AnswerRocket - https://www.linkedin.com/in/alon-goren-87889681/
• Pete Reilly, COO, AnswerRocket - https://www.linkedin.com/in/petereilly
• Shanti Greene, Head of Data Science and AI Innovation, AnswerRocket - https://www.linkedin.com/in/shantigreene
• Stew Chisam, Operating Partner, StellarIQ - https://www.linkedin.com/in/stewart-chisam-7242543/

Chapters:

00:00 Introduction

01:14 Sequoia's $10 Trillion AI AI Thesis

08:45 The Specialization Problem: From General Purpose to Actually Useful

16:56 Forward Deployed Engineers: Digging into the Palantir Model

22:47 The Intelligence Revolution vs. The Information Revolution

24:02 Prototyping and User Engagement

26:07 The Role of Business Analysts in AI Deployment

29:24 The Year of the Agent Reality Check: Replit and Autonomous Coding

35:09 Mock Data and Unit Test Cheats: Watch Out for These AI Coding Traps

#aiactually #enterpriseaiadoption #forwarddeployedengineers #palantirFDEmodel #AISpecialization #businessprocessautomation #ReplitAgent3 #AIScaffolding #EnterpriseIntegration #aiROI

The AI Actually crew tackles the pressing concerns keeping enterprise leaders up at night: shadow AI infiltrating organizations, the crucial distinction between machine learning and LLMs, and why context engineering matters more than prompt engineering. Jim Johnson takes the moderator chair, joined by regular Mike Finley and special guests Andy Sweet (Advanced Models Practice Lead) and Shanti Greene (Head of Data Science and AI Innovation).

Topics covered:

• Shadow AI: Why employees bypass IT to use personal AI tools
• Machine Learning vs LLMs: Understanding when each technology wins
• Prompt vs Context Engineering: Moving beyond moving commas
• Local vs cloud models: When ownership makes sense (and when it doesn't)
• Agent operations and the challenge of model stability
• The surprising costs of AI tokens, especially for audio applications

Follow the Gang

Jim Johnson, AnswerRocket, Managing Partner - https://www.linkedin.com/in/jim-johnson-bb82451/
Mike Finley, AnswerRocket, CTO - https://www.linkedin.com/in/mikefinley/ 

Andy Sweet, AnswerRocket, VP Enterprise AI Solutions - https://www.linkedin.com/in/andrewdsweet/
Shanti Greene, AnswerRocket, Head of Data Science and AI Innovation - https://www.linkedin.com/in/shantigreene/ 

Chapters:

00:00 Introduction to AI Actually and the Team

01:59 Kimi Model Release: Should We Care?

06:59 Shadow AI Definition and Enterprise Impact

12:08 Leveraging Machine Learning and LLMs Together

26:00 Prompt Engineering vs. Context Engineering

28:13 Using LLMs to Write Prompts

29:56 Memory and Agent Ops

33:05 AI Literacy and Context Engineering

34:27 Stability and Model Changes

37:28 The Harvard MBA Analogy for AI Agents

43:01 Local Open Source AI Models: Pros & Cons

49:23 The Real Costs of Enterprise AI

Keywords: shadow AI, machine learning vs LLMs, context engineering, prompt engineering, enterprise AI, local models, agent operations, AI costs, Kimi model, AI governance

The MIT study claiming 95% of AI pilots fail has everyone talking—but what's really behind these failures? In this episode, the AI, Actually crew tackles the hard truths about why enterprises struggle with AI implementations and what separates the toys from the tools. The conversation kicks off with AnswerRocket CEO Alon Goren sharing his journey from pre-PC era computing to building AI solutions that actually work.

The gang dives deep into the current state of AI agents in the workforce. Are they the new interns who need constant training, or are they about to trigger the largest labor disruption of our lifetime? From Klarna's famous (and failed) attempt to replace support staff to the reality of AI's impact on software development, the crew debates whether this time really is different.

Key topics covered:

• Why business problems, not technology, should drive AI adoption

• The real reason companies are freezing hiring (hint: it's not what you think)

• Meta's NBA-level salaries for AI talent and what it means for everyone else

• The "I wish" framework for identifying viable AI use cases • Why the best AI implementations start narrow and expand gradually

Chapters

00:00 Introduction: Meet Alon Goren

06:19 The AI Pilot Dilemma

11:57 Navigating AI Use Cases

21:08 AI Agents: The Largest Disruption of Our Lifetime? 32:05 How to Build Your AI Career

35:43 The AI Talent War: Recruitment and Retention Strategies

41:40 Assessing AI Depth of Expertise in Interviews