DevOps Daily with Fexingo: CI/CD, Kubernetes, and Modern Software Operations

When a single misbehaving mutating admission webhook in Kubernetes can take down your entire cluster, it's not just a developer headache—it's a production emergency. In this episode, Lucas and Luna break down the real-world cascade failure at a major fintech that lost $2.3 million in trading revenue over 47 minutes. They walk through how webhooks work, why a simple config map change triggered a chain reaction across 1,200 pods, and what the team did to recover—and prevent it from happening again. You'll learn about webhook timeout defaults, failure policy nuances, and why monitoring webhook latency matters more than you think. If you run Kubernetes in production, this episode might save your weekend.

#Kubernetes #MutatingAdmissionWebhook #CascadeFailure #K8sProduction #WebhookLatency #AdmissionController #CNCF #SiteReliabilityEngineering #IncidentResponse #CloudNative #K8sBestPractices #Fintech #PodSecurity #DevOps #Technology #FexingoBusiness #BusinessPodcast #Infrastructure

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily with Fexingo, Lucas and Luna tackle a frustratingly common Kubernetes pitfall: ResourceQuota silent denials. When a namespace's ResourceQuota is hit, the API server rejects resource creation without a clear error — often leaving teams debugging cryptic 'Internal error occurred' messages or watching pods hang in Pending. We walk through a real incident at a mid-size e-commerce company where a single over-quota Deployment caused cascading failures across 12 microservices. Lucas explains how kube-controller-manager calculates usage, why quota enforcement can silently block even `kubectl run`, and the exact kubectl commands to debug it. Luna shares a story about a team that accidentally set a quota on CPU but not memory, leading to runaway pods. We also cover how to set `kubectl describe quota` as a pre-deploy check, and why monitoring the `quota_exceeded` metric in Prometheus is a lifesaver. By the end, you'll know how to prevent your CI/CD pipelines from failing due to invisible quota walls.

#Kubernetes #ResourceQuota #DevOps #CI/CD #KubernetesTroubleshooting #Kubectl #Namespace #ClusterAdministration #ContainerOrchestration #CloudNative #InfrastructureAsCode #SiteReliabilityEngineering #PodLifecycle #Prometheus #Monitoring #Technology #FexingoBusiness #BusinessPodcast

Keep every episode free: buymeacoffee.com/fexingo

Episode 107 dives into a common misconfiguration: setting CPU limits on Kubernetes pods that unintentionally cap the Horizontal Pod Autoscaler's ability to scale. Lucas and Luna explore how the relationship between resource requests and limits affects the HPA algorithm, using a real-world example from a FinTech startup that saw scaling delays of over 40 seconds. They explain why the default CPU limit behavior can lead to throttling, how the HPA reads metrics from the `/metrics/resource/v1alpha1` endpoint, and why setting CPU limits too low can create a feedback loop where pods are throttled while the HPA waits for utilization to rise. The episode offers a concrete fix: using `targetCPUUtilizationPercentage` with appropriate request-to-limit ratios, and when to use `averageValue` instead of `averageUtilization`. No fluff, just one actionable lesson.

#Kubernetes #HPA #HorizontalPodAutoscaler #CPUThrottling #K8sScaling #DevOps #FinTech #ResourceManagement #CloudNative #CI/CD #PodAutoscaling #KubernetesLimits #ProductionTroubleshooting #SiteReliability #FexingoBusiness #BusinessPodcast #TechOps #ContainerOrchestration

Keep every episode free: buymeacoffee.com/fexingo

In this episode, Lucas and Luna dive into a counterintuitive Kubernetes failure mode: correctly configured PodDisruptionBudgets that actually create cascading availability failures. Using a real-world example from a large e-commerce platform migrating to spot instances in July 2026, they explain how PDBs interact with cluster autoscaler taint-based evictions and node drain logic to deadlock workloads. Lucas breaks down the specific scenario — a 5-replica deployment with maxUnavailable=1 that should be safe but stalls an entire node pool — and walks through the underlying mechanism: how Kubernetes honors PDBs during voluntary disruptions but blocks evictions when nodes are cordoned, leading to hung pods and scaling failures. Luna brings data from a recent CNCF survey showing 40% of teams using spot instances have encountered this exact pattern. They cover mitigation strategies including pod-level priority, descheduler policies, and PDB monitoring, and discuss why this becomes more acute as GPU spot instances gain traction. Practical, specific, and immediately actionable for any SRE running Kubernetes at scale.

#Kubernetes #PodDisruptionBudget #SpotInstances #ClusterAutoscaler #NodeDrain #K8sFailureModes #SiteReliabilityEngineering #CloudNative #CNCF #GPUSpot #TaintBasedEvictions #MaxUnavailable #PodPriority #Descheduler #K8sScaling #DevOps #Technology #FexingoBusiness

Keep every episode free: buymeacoffee.com/fexingo

Episode 105 of DevOps Daily dives into a surprising CI bottleneck: Docker image layer caching that backfires. Lucas and Luna examine how layer reuse assumptions break down in environments with dynamic dependency trees, leading to cache-miss avalanches and pipeline slowdowns. They walk through a real case study from a mid-size e-commerce team that saw CI times balloon from 8 to 22 minutes after adopting multi-stage builds, then cut them back to 11 minutes by switching to a cache-unfriendly base image strategy. The episode covers cache hit ratios, layer invalidation patterns, and why buildkit's --cache-from isn't always the answer. No fluff, just a sharp technical breakdown.

#Docker #ImageLayerCaching #CI #Buildkit #MultiStageBuilds #CacheMiss #PipelinePerformance #DevOps #SoftwareOperations #Technology #FexingoBusiness #BusinessPodcast #LucasAndLuna #DependencyManagement #CacheHitRatio #LayerInvalidation #BaseImageStrategy #BuildOptimization

Keep every episode free: buymeacoffee.com/fexingo

Lucas and Luna dissect a specific CI/CD failure pattern: Docker image layer caching that actually degrades pipeline performance. Using a real-world example of a mid-stage startup whose builds went from 4 minutes to 22 minutes after adopting a naive cache strategy, they explain layer invalidation, cache-miss cascades, and the trade-off between cache reuse and build determinism. Lucas walks through why copying package.json separately from source code matters, and Luna challenges the assumption that more cache always means faster builds. They end by discussing when to bypass the cache entirely for critical security patches. No abstract theory — just a concrete debugging story any team running containerized pipelines will recognize.

#Docker #CICD #ImageLayerCaching #BuildPerformance #CI/CD #DevOps #FexingoBusiness #BusinessPodcast #Technology #SoftwareEngineering #Containerization #PipelineOptimization #LayerInvalidation #CacheMiss #BuildTime #Dockerfile #PackageJson #SecurityPatches

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily, Lucas and Luna dive into a subtle but dangerous Kubernetes configuration trap: pod overhead misassignment. When cluster operators set pod overhead values incorrectly—especially in clusters running multiple sandboxed runtimes like Kata Containers or gVisor—the kubelet can overcommit node resources, leading to unexplained OOM kills and node pressure. Lucas walks through a real incident where a team lost three nodes in production because their pod overhead settings didn't account for per-pause-container memory. Luna pushes for practical guardrails, and they discuss how to validate overhead with node-level metrics and resource quotas. A must-listen for anyone managing multi-tenant clusters or experimenting with runtime-class isolation.

#Kubernetes #PodOverhead #ResourceExhaustion #KataContainers #gVisor #RuntimeClass #OOMKill #NodePressure #DevOps #CloudNative #ContainerOrchestration #Troubleshooting #Infrastructure #Technology #DevOpsDaily #FexingoBusiness #BusinessPodcast #SRE

Keep every episode free: buymeacoffee.com/fexingo

Episode 102 of DevOps Daily dives into a lesser-known risk with Kubernetes Vertical Pod Autoscaler (VPA): its ability to cause cascading instability in production workloads. Lucas and Luna walk through a real-world case where VPA's recommender triggered simultaneous resource resizing across multiple microservices, leading to pod evictions, request timeouts, and a multi-hour incident at a mid-sized SaaS company. They explain VPA's resource recommendation algorithm, why its default mode forces pod restarts, and how a simple configuration change — setting updateMode to 'Off' — turned VPA from a destabilizing force into a safe guidance tool. The episode also covers how to pair VPA with Horizontal Pod Autoscaler (HPA) without conflict, key metrics to monitor (pod restart rate, P99 latency), and a practical workflow: run VPA in recommendation-only mode, review changes during off-peak hours, apply manually. Perfect for teams already using HPA and wondering if VPA is worth the complexity — spoiler: it is, but only with guardrails.

#Kubernetes #VerticalPodAutoscaler #DevOps #PodAutoscaling #K8sReliability #SiteReliabilityEngineering #CloudNative #ContainerOrchestration #PodEvictions #ResourceManagement #MicroservicesStability #IncidentResponse #SaaSPlatforms #HPA #VPA #FexingoBusiness #Technology #DevOpsDaily

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily, Lucas and Luna dig into a subtle but impactful Kubernetes networking issue: how kube-proxy's default iptables mode can cause periodic latency spikes under moderate traffic. They walk through the mechanics of iptables rule evaluation, how conntrack table flushes trigger cascading latency, and why large clusters with frequent Service updates are especially vulnerable. They also discuss real-world incidents, including a case where a 500-node cluster saw 99th-percentile API response times jump from 10ms to over 500ms during a Service rollout. The conversation covers mitigation strategies like switching to IPVS mode, tuning conntrack parameters, and using endpoint slice batching. A must-listen for anyone running Kubernetes at scale who wants to understand a silent performance killer.

#Kubernetes #kube-proxy #iptables #NetworkLatency #Conntrack #IPVS #DevOps #CloudNative #SRE #ClusterPerformance #ServiceMesh #EndpointSlice #K8sNetworking #LatencySpikes #Technology #SoftwareOperations #FexingoBusiness #BusinessPodcast

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily, Lucas and Luna mark episode 100 by tackling a problem that haunts every multi-team cluster: cost allocation. Lucas explains how Kubernetes's shared infrastructure model makes it nearly impossible to attribute spending accurately to teams, namespaces, or workloads without a purpose-built metrics layer. He breaks down the three common pitfalls — relying on namespace-level averages, ignoring pod resource wastage, and missing spot-instance price volatility — and contrasts a hypothetical team's naive approach (using kubectl top and cloud console totals) against a mature implementation using OpenCost with real-time data from Prometheus. Luna chimes in with a story about a startup that doubled its cloud bill overnight because a single team deployed CPU-inefficient inference pods. The episode ends with a forward-looking question about whether the Kubernetes community will eventually bake cost visibility into the control plane itself.

#Kubernetes #CostAllocation #CloudCost #FinOps #OpenCost #KubernetesCostManagement #DevOps #Technology #FexingoBusiness #BusinessPodcast #Podcast #DevOpsDaily #KubernetesTips #CloudInfrastructure #CostOptimization #Kubectl #Prometheus #SpotInstances

Keep every episode free: buymeacoffee.com/fexingo