Is your Kubernetes cluster ready for the next generation of AI workloads? Join Buoyant Tech Evangelist Flynn and Red Hat's Shane Utt from the AI Gateway Working Group as they reveal the critical shift from header-based to payload-based Kubernetes Networking for Agentic AI.

In this episode of The Kubernetes AI Show, we dive deep into the challenges of running AI workloads on cloud native infrastructure with AI Gateway leaders, Flynn and Shane. They discuss how the AI Gateway Working Group evolved from the Gateway API inference extension (GIE) to focus on the average Kubernetes user who is often performing inference via egress outside the cluster. A key challenge is that AI models are fundamentally different from microservices—they are not interchangeable, making advanced routing a must.

The conversation highlights a critical shift: networking infrastructure must now be retrofitted for payload processing for routing to ensure security, compliance (like GDPR), and proper model selection. We explore parallels between AI’s hardware demands and High Performance Computing (HPC), revealing the problem of low LLM hardware utilization and the talent gap between data scientists and network engineers. Finally, the co-leads detail the security implications of Agentic AI's autonomy, advocating for a defense-in-depth security strategy that includes multi-cluster routing and Model Context Protocol (MCP) servers with 'elicitation' to prevent accidental, destructive actions.

Read the blog post: www.buoyant.io/ai-kubernetes-episode/agentic-ai-security-on-kubernetes-understanding-payload-routing-and-defense-in-depth

Follow us on LinkedIn: https://www.linkedin.com/company/the-ai-kubernetes-show/

Key Learnings 

✓ Learn why AI models require advanced, non-interchangeable routing logic compared to standard microservices.

✓ Discover how the shift to payload processing is essential for semantic routing, security, and GDPR compliance.

✓ Understand why AI’s low hardware utilization (20-30%) is a known problem from High Performance Computing (HPC).

✓ See how Agentic AI introduces a "deeply terrifying" security risk that demands a defense-in-depth security approach like MCP elicitation.

If you're building or securing AI workloads on Kubernetes, you can't afford to miss this discussion! Subscribe to The Kubernetes AI Show for more foundational insights, like this one from the AI Gateway leaders. 

Hit the like button and let us know in the comments: What is the most critical challenge you are facing when building Kubernetes Networking for your Agentic AI applications?

#KubernetesNetworking #AIGateway #AgenticAI #CloudNative #GatewayAPI #LLMs #K8s #Microservices #TechEvangelist

Is your organization ready for the rapid pace of AI adoption? We sat down with Rob Koch (Slalom Build, CNCF Deaf and Hard of Hearing Working Group Co-Chair) to discuss how engineers can master automation in the age of non-deterministic systems, focusing on real-world use cases in Kubernetes and platform engineering.

In this episode of the AI Kubernetes Show, Koch provides essential insights on building a robust strategy for AI adoption. He emphasizes moving beyond a "move fast, fail fast" approach to focus on clear outcomes, which he describes as applying the engineering mindset to AI. A key discussion point is mitigating risk in non-deterministic systems by implementing strong guardrails, constraints, and human oversight—often by providing precise context and clear specifications.

Koch shares a successful example of automating repetitive tasks using AI sub-agents in a database upgrade project. The system analyzed logs, diagnosed resource needs (checking for over/under-provisioning), and automatically drafted documentation, aligning with the "Don't Repeat Yourself" (DRY) principle. Additionally, he highlights the transformative power of AI for accessibility and the promise of LLMs in sign language recognition. 

Read the blog post: www.buoyant.io/ai-kubernetes-episode/the-engineering-mindset-for-ai-from-automation-to-guardrails 

Follow us on LinkedIn: https://www.linkedin.com/company/the-ai-kubernetes-show/

Takeaways

✓ Strategic AI adoption starts with defining the desired outcome and working backward, not a "fail fast" approach.

✓ Mitigate risk in non-deterministic AI systems by implementing guardrails, human oversight, and limiting context.

✓ AI is best suited for automating repetitive tasks (DRY principle) to free up engineering time.

✓ Subject matter expertise remains crucial for verifying AI output and detecting "hallucinations."

✓ AI is a powerful tool for accessibility, helping bridge communication gaps (e.g., English syntax correction for ASL users).

If you're tackling AI implementation or just getting started, hit the like button and subscribe for more deep dives into platform engineering! What is the biggest challenge your team faces when adopting AI? Let us know in the comments below! 

#AI #Kubernetes #PlatformEngineering #AIAutomation #Accessibility 

The rise of AI has dramatically increased code velocity. Join us as Comcast Platform DevOps Engineer Curtis Cook breaks down the shift to platform-level controls and a "zero trust" mindset to secure the future of Platform Engineering. 

In this episode of the AI Kubernetes show, Curtis Cook shares his expert insights on how the accelerating pace of code development driven by AI tooling is forcing platform teams to completely rethink code quality and security management. He explains that the massive amount of AI-generated code is like "hiring thousands of junior developers," creating a significant scaling problem. Drawing lessons from Kubernetes, Comcast is implementing platform-level controls to automate mundane tasks and enforce standards, adopting the cultural shift of "guardrails, not gates."

Curtis dives into Security in the age of AI, where the concern moves beyond infrastructure because AI actively makes decisions, making securing AI workloads absolutely crucial. Comcast’s strategy involves a "zero trust by default" mindset and applying the least privileged access model to AI agents to combat risks like mitigating hallucinations and "context bloat." In the section "The non-deterministic world of AI," he discusses the profound change presented by non-deterministic AI outputs and security threats like prompt injection and model drift. This shift requires new testing methods like statistical validation instead of traditional unit tests. Curtis also highlights the importance of the CNCF community for aspiring platform engineers, noting how AI can help newcomers learn Kubernetes fundamentals.

Find all resources and more in the blog post: 

Takeaways

✓ The increase in code velocity from AI-generated code requires a move from policing individual commits to implementing robust platform-level controls.

✓ AI is treated as a critical system at Comcast, secured with a "zero trust by default" mindset.

✓ Mitigating hallucinations and context bloat is achieved by applying the least privileged access model to AI tools.

✓ Non-deterministic AI systems introduce security threats and require advanced testing methods like statistical validation and confidence scoring.

✓ The Kubernetes and CNCF community is essential for platform engineers, and AI-powered tools can help new members get started.

What is your biggest security concern with generative AI in your software development lifecycle? Let us know in the comments below! 

If you found this discussion valuable, please like this video, share it with your team, and subscribe for more deep dives into Platform Engineering and cloud-native technology! 

#PlatformEngineering #GenerativeAI #Kubernetes #ComcastTech #CNCF #ZeroTrust #AIsecurity #CodeVelocity #DevOps 

The future of the service mesh is here! Don't miss this deep dive with Linkerd creator Oliver Gould on how to conquer the toughest challenges of running stateful AI workloads in production.

In this episode of the AI Kubernetes Show, Buoyant CTO Oliver Gould reveals how Linkerd is adapting its battle-tested features to the new demands of the AI workloads era. He emphasizes that the cost of AI inference failures is extremely high, making network layer tools like Linkerd’s intelligent load balancing and fault tolerance features more critical than ever. The discussion zeroes in on the emerging MCP (Model Context Protocol), a stateful streaming protocol that poses unique challenges to traditional network tooling, particularly because critical information is buried in the payload, not the headers.

Gould details the clash between MCP’s stateful nature and the prevalent Kubernetes microservice architecture, stressing the need to evolve infrastructure planning for auto-scaling and resource provisioning for GPUs. He shares a vision for Linkerd’s MCP support, which includes a load balancer mode optimized for streams and extending policy APIs for MCPRoute. Finally, Gould touches on the positive impact of AI tooling on developer productivity, sharing a personal anecdote about how Copilot and Cortex dramatically accelerated the diagnosis of a complex Go race condition, proving the value of using robots for data analysis and humans for high-value design.

Read the blog post: 

Key Learnings/Takeaways

✓ The high cost of AI inference failures makes network reliability (load balancing, retries) an enormous "cost lever" for organizations.

✓ MCP is a stateful streaming protocol where success/failure information is in the JSON payload, making it opaque to most network headers-based tooling.

✓ Running stateful MCP workloads at scale requires evolving stateless Kubernetes infrastructure around auto-scaling, GPU provisioning, and stable load distribution.

✓ Linkerd is building a load balancer optimized for streams and extending policy APIs (like MCPRoute) to support the new protocol.

✓ AI tooling, like Copilot and Cortex, significantly boosts developer productivity by automating boilerplate (like YAML) and quickly diagnosing complex issues.

If you found this discussion valuable, hit the Like button and Subscribe for more insights from the AI Kubernetes Show! Let us know in the comments: What is the biggest stateful workload challenge you are currently facing in your Kubernetes cluster?

#Linkerd #ServiceMesh #Kubernetes #MCP  #LoadBalancing #CloudNative #KubeCon

Don't miss this deep dive with Bloomberg's Alexa Griffith on how a financial firm is navigating the cultural and technological shift to Generative AI. Learn the critical role of platform engineering and Kubernetes in building a secure, scalable, and future-proof AI infrastructure.

In this episode of the AI Kubernetes Show, Senior Software Engineer Alexa Griffith discusses Bloomberg's journey from predictive to Generative AI. She reveals how their early investment in Kubernetes back in 2016 for AI workloads led to open-source projects like KServe and the newer Envoy AI Gateway, built to manage traffic and unify APIs in a hybrid cloud environment. The conversation explores the massive challenge of wrangling the non-deterministic nature of LLMs, which dramatically affects cost and observability—a problem Bloomberg addresses by building a "deterministic cage" around the core models.

Alexa details the new requirements for platform engineering teams, including the shift to token streaming and increased GPU usage. She introduces the concept of a Model Garden as a crucial layer of abstraction and enablement, providing developers with benchmarking data and features to make informed, use-case-driven choices. The discussion also covers critical security concerns, the use of LLMs as judges for evaluation, and the debunking of assumptions around RAG systems safety. Finally, she explains the evolution of performance metrics, highlighting the importance of time to first token and token flow for a good user experience in the era of agentic systems.

Read the blog post: 

Key Learnings/Takeaways

✓ Generative AI is an evolution, not a revolution, built on a strong predictive AI foundation.

✓ Non-deterministic LLMs are managed by building a "deterministic cage" for control, cost, and observability.

✓ The Model Garden is a key platform engineering concept for abstracting model complexity and empowering developers.

✓ New performance metrics are critical, specifically time to first token (snappiness) and token flow (continuous experience).

✓ The Envoy AI Gateway and the MCP protocol are essential for unifying disparate model APIs and managing agentic systems.

If you enjoyed this conversation, hit the like button, subscribe for more content on Kubernetes and AI, and leave a comment below! What is the biggest platform engineering challenge you're currently facing with Generative AI?

#GenerativeAI #AI #Kubernetes #PlatformEngineering #LLMOps #AIInfrastructure #KServe #CNCF #CloudNative #TechAtBloomberg

In this episode of The AI Kubernetes Show, we talked with Troy Connor, Senior Software Engineer at Microsoft and a maintainer of Porter and Kubernetes controller-runtime maintainer (two CNCF projects), about the balance between enterprise work and open source contribution and the impact of AI on software development and community maintenance. This post summarizes our discussion, live from KubeCon in Atlanta.

On this episode of The AI Kubernetes Show, we chatted with Diana Todea, Developer Experience Engineer at Victoria Metrics and Co-chair for the CNCF Neurodiversity Working Group. This post summarizes our discussion, which covered the impact of AI, generational shifts, and the role of Developer Experience (DevX).

In this AI Kubernetes Show episode, we chat with OpenSSF CTO and Linux Foundation Chief Security Architect Christopher Robinson, or CRob, as everyone calls him. We explore the potential and the inherent risks of integrating AI into the open source security landscape. 

Learn how structured observability data and the platform team enable fast AI troubleshooting. Balance AI automation with essential developer craftsmanship.

Learn how prompt engineering and a cutting-edge multi-agent workflow pipeline are reshaping software dev. Get the framework for safe AI integration.