DevOps Daily with Fexingo: CI/CD, Kubernetes, and Modern Software Operations

In DevOps Daily Episode 99, Lucas and Luna dive into a dangerous and often overlooked Kubernetes behavior: cascading deletes of namespaces and the resource storms they trigger. Using a real incident at a mid-size SaaS company that accidentally deleted a production namespace, they explain how `kubectl delete namespace` doesn't just remove pods—it tears down every resource in that namespace in parallel, causing massive API server load, etcd latency spikes, and even cross-namespace side effects. They walk through the concrete failure cascade: from a namespace deletion to a 5-second etcd P99 latency increase, to a 3-minute control plane unavailability. They also cover mitigation strategies like using finalizers on critical namespaces, implementing namespace deletion prevention with admission webhooks, and setting up etcd request rate limiting. This episode gives DevOps engineers a specific, actionable understanding of a hazard rarely covered in training materials.

#Kubernetes #NamespaceDeletion #etcd #APIServer #DevOps #SiteReliabilityEngineering #CloudNative #KubernetesBestPractices #AdmissionWebhook #Finalizers #ControlPlane #ProductionIncident #ResourceStorm #KubernetesFailure #KubernetesTip #Technology #FexingoBusiness #BusinessPodcast

Keep every episode free: buymeacoffee.com/fexingo

Lucas and Luna dive into a nasty Kubernetes failure pattern that catches teams running GPU workloads: how the Cluster Autoscaler can't handle GPU node pools when pods request partial GPU resources. They break down a real incident at a mid-size AI startup where 12 nodes drained in 8 minutes, trace the root cause to the autoscaler's inability to aggregate fractional GPU requests across nodes, and walk through the workaround — using nodeSelector or taints to pin GPU pods to dedicated node pools. They also discuss how the upstream Kubernetes community is approaching a fix with the proposed 'group scheduling' feature in Cluster Autoscaler v2. Practical and specific, this episode gives listeners a concrete pattern to avoid and a monitoring metric to watch.

#Kubernetes #ClusterAutoscaler #GPUNodes #K8sFailures #Infrastructure #CloudNative #DevOps #AIWorkloads #NodePools #Scheduling #K8sTroubleshooting #CloudComputing #PodAutoscaling #TechOps #FexingoBusiness #BusinessPodcast #Technology #K8sScaling

Keep every episode free: buymeacoffee.com/fexingo

Lucas and Luna dive into a surprisingly common Kubernetes failure: MutatingAdmissionWebhooks that silently time out under load, bringing cluster-wide latency to a crawl. They walk through a real incident at a mid-size fintech company where a single webhook — a simple config-injection sidecar — started timing out at around 60 requests per second, causing every API call to pile up. They explain why Kubernetes doesn't enforce a hard timeout by default, how the API server queues webhook responses, and why your observability stack likely won't catch this unless you're monitoring API server request duration at the verb level. They also cover the fix: setting a proper failurePolicy and timeoutSeconds, plus a hint about using webhook-side caching. If you run any admission webhook in production, this one hits close to home.

#Kubernetes #MutatingAdmissionWebhook #ClusterLatency #SiteReliabilityEngineering #K8sIncidents #AdmissionController #APIServerPerformance #CloudNativeInfrastructure #DevOps #Technology #FexingoBusiness #BusinessPodcast #Fintech #KubernetesTroubleshooting #WebhookTimeout #K8sFailureModes #ProductionOutage #KubernetesBestPractices

Keep every episode free: buymeacoffee.com/fexingo

Episode 96 of DevOps Daily with Fexingo dives into a subtle but dangerous Kubernetes failure pattern: taint-based evictions silently blocking node drains. Lucas and Luna break down a real incident where a cluster-wide taint added by an automation tool prevented nodes from draining during a planned maintenance window, causing cascading pod failures and a near-P0. They explain how taint propagation works, why taint-based evictions bypass PodDisruptionBudgets, and what operators can do to avoid the trap. The episode includes a concrete example from a mid-size e-commerce platform running on EKS. No clickbait, just actionable ops insight.

#Kubernetes #DevOps #NodeDrain #TaintBasedEvictions #K8sFailure #PodDisruptionBudget #ClusterOps #EKS #CloudNative #Infrastructure #SRE #K8sTroubleshooting #Technology #FexingoBusiness #BusinessPodcast #DevOpsDaily #K8sBestPractices #K8sAutomation

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily with Fexingo, Lucas and Luna dive into a specific gotcha in the Kubernetes PodSecurityPolicy (PSP) migration to Pod Security Admission (PSA). They walk through a real scenario where enabling PSA in audit mode silently suppresses warnings for privileged pods that were previously blocked by PSP. Lucas explains the mechanics: PSP evaluates at admission and blocks; PSA in audit mode logs violations but allows the pod through, creating a security blind spot. They discuss why teams migrating from PSP to PSA often miss this shift in enforcement, how audit-mode logs can be drowned out by cluster noise, and a concrete remediation: using custom ValidatingAdmissionPolicies to restore blocking behavior before fully retiring PSP. Luna shares a story of a fintech team that discovered this gap during a compliance audit six months post-migration. The episode closes with a forward-looking note on the future of policy enforcement and the OPA Gatekeeper ecosystem.

#Kubernetes #PodSecurityPolicy #PodSecurityAdmission #DevOps #CloudNative #Security #K8sPolicy #ClusterSecurity #PSPMigration #AuditMode #ValidatingAdmissionPolicy #OPAGatekeeper #Technology #SoftwareOperations #FexingoBusiness #BusinessPodcast #DevOpsDaily #K8sBestPractices

Keep every episode free: buymeacoffee.com/fexingo

Episode 94 of DevOps Daily with Fexingo dives into a subtle but dangerous Kubernetes issue: projected ServiceAccount token expiration causing CronJobs to fail silently. Lucas and Luna walk through a real case where a team's nightly batch processes started failing at 3 AM with no clear error, traced back to tokens expiring mid-job after a cluster upgrade. They explain how bound service account tokens work, why the default one-hour expiry can break long-running CronJobs, and what to configure in the Pod spec — including the 'expirationSeconds' field and token mounting paths. The episode also covers how to test token expiry locally using 'kubectl debug' and how to set up Prometheus alerts for token-related failures. Listeners walk away with a concrete checklist: audit your CronJob Pod specs, set 'expirationSeconds' to match your longest job duration, and monitor token renewal logs.

#Kubernetes #DevOps #CronJob #ServiceAccount #TokenExpiration #CloudNative #SiteReliability #K8sSecurity #PodSpec #BatchJobs #Prometheus #Monitoring #Kubectl #ContainerOrchestration #InfrastructureAsCode #Technology #FexingoBusiness #BusinessPodcast

Keep every episode free: buymeacoffee.com/fexingo

Episode 93 of DevOps Daily drills into a subtle Kubernetes scheduling failure: pod priority inversion. Lucas and Luna walk through a real-world scenario where a high-priority DaemonSet preempts critical batch jobs, leading to node saturation and cascading evictions. They explain why the default priority class hierarchy is misleading, how the preemption algorithm selects victims, and why monitoring only CPU and memory misses the real bottleneck. Specific data from a 200-node cluster shows that fixing priority inversion reduced scheduling delays by 38 percent. Practical guidance for auditing priority classes and setting correct preemption policies.

#Kubernetes #Scheduling #PodPriority #PriorityInversion #NodeSaturation #Preemption #DaemonSet #BatchJobs #ClusterAutoscaler #Descheduler #SchedulerProfiling #QualityOfService #PriorityClass #ProductionKubernetes #CNCF #Technology #DevOps #FexingoBusiness

Keep every episode free: buymeacoffee.com/fexingo

Episode 92 of DevOps Daily dives into a dangerous blind spot in Kubernetes cluster operations: the default Cluster Autoscaler's total disregard for spot instance interruption notices. Lucas and Luna break down how AWS, Azure, and GCP issue two-minute warnings before reclaiming spot capacity, yet the Cluster Autoscaler treats those nodes as healthy until they're actually terminated. The result? Pods get forcibly evicted with no time to drain, causing cascading failures in stateful workloads and batch jobs. Lucas explains a concrete fix: a custom interrupt-handling webhook that taints the node the instant a notice arrives, giving the scheduler a full 120 seconds to reschedule pods gracefully. They also walk through a real-world case where a fintech company lost 12 percent of their inference requests during a spot reclaim storm because they relied on the default behavior. If you run Kubernetes on spot instances — and 60 percent of surveyed clusters do in 2026 — this episode will save you from a painful outage.

#Kubernetes #ClusterAutoscaler #SpotInstances #CloudNative #DevOps #AWS #Azure #GCP #PodEviction #NodeDrain #Infrastructure #SRE #ContainerOrchestration #Resilience #CloudCost #FexingoBusiness #TechnologyPodcast #DevOpsDaily

Keep every episode free: buymeacoffee.com/fexingo

Kubernetes cluster autoscaler is a workhorse for dynamic workloads, but when you're running spot (preemptible) instances, the node autoscaler's default behavior can actually hurt reliability. In this episode, Lucas and Luna break down a real incident from a large ad-tech platform running 2,000+ nodes on AWS EC2 Spot Instances. When spot interruption rates spiked during a regional AZ outage, the cluster autoscaler's rigid scale-down logic kept terminating nodes that were seconds away from stabilizing, causing a cascading re-scheduling loop that nearly took down their real-time bidding service. They walk through the specific configuration failures: the unbound scale-down-delay-after-add, the lack of node disruption budgets for spot pools, and the missing mixed-instance-policy fallback. Plus, they share the three YAML changes that fixed it — including custom priority expander logic and a proactive node-drain webhook. If you're running spot nodes in production, this is the episode that will save you from a 2 AM pager.

#Kubernetes #ClusterAutoscaler #SpotInstances #AWS #EC2 #NodeScaling #RealTimeBidding #AdTech #DevOps #SiteReliability #K8sFailures #InfrastructureAsCode #Technology #Podcast #FexingoBusiness #BusinessPodcast #DevOpsDaily #CloudOperations

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily with Fexingo, Lucas and Luna dive into a subtle but dangerous Kubernetes behavior: ConfigMap updates to ingress controllers can trigger memory pressure across worker nodes. They walk through a real incident where a team's single ConfigMap change caused a cascade of OOM kills across 12 nodes in a staging cluster. You'll learn why ingress controllers like NGINX reload their entire configuration on every ConfigMap update, how that reload spikes memory usage, and what mitigations teams can adopt — from using reloader sidecars to structuring ConfigMaps by namespace. A focused, actionable episode for anyone running production Kubernetes.

#Kubernetes #IngressController #ConfigMap #MemoryPressure #NGINX #OOMKill #ClusterStability #DevOps #SRE #CloudNative #CNCF #PodDisruption #NodeMemory #FexingoBusiness #BusinessPodcast #Technology #K8sTips #ProductionIncidents

Keep every episode free: buymeacoffee.com/fexingo