DevOps Daily with Fexingo: CI/CD, Kubernetes, and Modern Software Operations

In this episode of DevOps Daily, Lucas and Luna dig into a surprisingly common Kubernetes headache: namespaces stuck in the Terminating state because of object finalizers. They walk through a real incident where a single stuck custom resource with a finalizer held up an entire namespace deletion, blocking a whole environment teardown and causing pipeline failures. You'll learn what finalizers actually do, how they can silently break cluster cleanup, and the step-by-step debugging approach — from checking finalizers with kubectl to patching or force-deleting when needed. They also discuss safe alternatives and when it's better to fix the controller instead of bypassing the safety mechanism. This episode gives you a concrete mental model and a few kubectl commands you'll actually use in production.

#KubernetesFinalizers #NamespaceDeletion #K8sOperators #CloudNative #DevOps #KubectlTips #ClusterCleanup #KubernetesTroubleshooting #ControllerDesign #CustomResourceDefinitions #TechPodcast #FexingoBusiness #BusinessPodcast #Technology #SoftwareEngineering #InfrastructureAsCode #SiteReliabilityEngineering #CloudInfrastructure

Keep every episode free: buymeacoffee.com/fexingo

When a Kubernetes node reboots for a kernel update or hardware maintenance, pods don't just vanish and come back cleanly. The kubelet marks them as terminated, the container runtime loses state, and what looks like a routine maintenance window turns into failed readiness probes, stuck terminating pods, and a cascade of restarts. In episode 158 of DevOps Daily, Lucas and Luna break down the actual mechanics of node reboot behavior: how the kubelet handles graceful shutdown, why container runtimes like containerd and CRI-O can orphan processes, and why your deployment's restartPolicy is the hidden variable that decides whether that reboot is a blip or a production incident. They also walk through a real scenario where a node reboot during off-peak hours caused a payment service to restart with cold caches, triggering a 5x latency spike. If you've ever wondered why your pods don't come back the way they left, or why a simple reboot feels like playing roulette with your uptime, this episode gives you the mental model and the practical mitigations: preStop hooks, terminationGracePeriodSeconds, and podDisruptionBudgets that actually work.

#Kubernetes #NodeReboot #Kubelet #ContainerRuntime #PodRestart #DevOps #SRE #CloudNative #Containerd #CRIO #PreStopHook #TerminationGracePeriod #PodDisruptionBudget #SiteReliability #Infrastructure #Tech #FexingoBusiness #BusinessPodcast

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily, Lucas and Luna dig into a silent killer in Kubernetes clusters: systemd journal rotation. When the kubelet's logs are rotated out before a crash is captured, debugging becomes guesswork. They walk through the mechanics of journald's size and time-based rotation, the systemd-journald daemon's default rate limits, and how a busy cluster can lose critical kubelet logs in seconds. They share a real-world scenario where a node crash loop went unnoticed for hours because the logs were gone. They also discuss practical fixes: setting SystemMaxUse and RuntimeMaxUse, adjusting RateLimitIntervalSec, forwarding logs to a central aggregator, and using node problem detector to catch issues early. By the end, you'll know exactly how to ensure your kubelet logs survive the next incident.

#Kubernetes #DevOps #Systemd #Journald #LogRotation #Kubelet #CrashLoop #Troubleshooting #SiteReliability #CloudNative #Observability #Logging #NodeProblemDetector #IncidentResponse #TechPodcast #Business #FexingoBusiness #Podcast

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily, Lucas and Luna dig into a classic Kubernetes failure mode: kubelet eviction signals that silently kill pods. The trigger is often memory pressure on the node, and the fallout is worse than a crash — the pod vanishes without a proper termination, leaving no time for graceful shutdown. They walk through a real-world scenario from an e-commerce platform where the memory limit was set too high, the node ran out of memory, and the eviction manager picked victims in an order nobody expected. Lucas explains the difference between soft and hard eviction thresholds, and why the kubelet's eviction signal isn't just about memory — it can also be driven by disk pressure, inodes, or even PID limits. They also cover the critical role of the eviction manager's ranking by QoS class, and why Burstable pods often get hit first. The conversation wraps with practical advice: monitor eviction signals early, set realistic memory requests, and test your graceful shutdown path with real load. If you've ever lost a pod without a clear reason, this episode gives you the mental model to troubleshoot it fast.

#KubeletEviction #Kubernetes #PodDeaths #MemoryPressure #EvictionManager #QoSClasses #BurstablePods #GracefulShutdown #NodePressure #DevOps #SRE #CloudNative #ContainerOrchestration #TechPodcast #FexingoBusiness #BusinessPodcast #SoftwareOperations #CI/CD

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily, Lucas and Luna dive into the hidden pitfalls of Horizontal Pod Autoscaler (HPA) with custom metrics. They explore a real-world incident where a stale external metric from a monitoring system caused the HPA to scale down a critical service during a traffic spike, leading to degraded performance and a cascade of alerts. The discussion covers how HPA evaluates metrics, the dangers of metric staleness and missing data, and practical strategies to prevent such failures—like setting proper thresholds, using multiple metrics, and implementing safeguards like the stabilization window and scale-down policies. Tune in to learn how to keep your autoscaling reliable and avoid the 'zombie metric' trap.

#Kubernetes #HPA #Autoscaling #CustomMetrics #DevOps #SRE #Monitoring #Prometheus #CloudNative #Infrastructure #Scaling #Reliability #Microservices #ContainerOrchestration #TechPodcast #FexingoBusiness #BusinessPodcast #DevOpsDaily

Keep every episode free: buymeacoffee.com/fexingo

In this episode, Lucas and Luna dive into a frequent but often misunderstood Kubernetes performance issue: slow cluster DNS queries. They unpack how CoreDNS, the default DNS server, can become a bottleneck when node-level caching is missing, and why adding NodeLocal DNSCache often resolves latency spikes. With a concrete example from a payments company that saw p99 DNS latency drop from 120ms to under 5ms, they explain the mechanics behind the fix, including iptables rules, conntrack limits, and UDP timeouts. Listeners will learn practical tuning knobs, such as setting the right cache sizes and avoiding common misconfigurations like setting 'singleStack' to true when dual-stack is needed. The hosts also touch on when jumping straight to a service mesh or eBPF-based solution might be overkill. This episode is ideal for platform engineers and SREs who want to make their clusters faster without adding complexity.

#Kubernetes #CoreDNS #NodeLocalDNSCache #DNSPerformance #ClusterNetworking #SiteReliabilityEngineering #PlatformEngineering #CloudNative #DevOps #TechPodcast #FexingoBusiness #BusinessPodcast #Infrastructure #Monitoring #PerformanceOptimization #Conntrack #Iptables #UDPTimeout

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily, Lucas and Luna dive into a subtle but impactful Kubernetes issue: CPU manager policies. They explore how the default 'none' policy can lead to unpredictable performance due to CPU cache contention and context switching, and how the 'static' policy can guarantee exclusive cores but introduces its own pitfalls, like the risk of pods being stuck on specific nodes. Using a real-world example of a latency-sensitive payment service that saw p99 latency spikes after scaling, they explain the trade-offs and offer practical advice on when to enable static CPU management. They also touch on the interaction with autoscaling and how to monitor for CPU manager-related issues. A must-listen for anyone running performance-critical workloads on Kubernetes.

#Kubernetes #CPUManager #Performance #DevOps #CloudNative #ContainerOrchestration #PodScheduling #Latency #CI/CD #Technology #Business #FexingoBusiness #BusinessPodcast #DevOpsDaily #Kubelet #StaticPolicy #CPUAffinity #PerformanceTuning

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily, Lucas and Luna dig into Kubernetes Pod Disruption Budgets and why they are a common culprit behind node drain stalls. They walk through a real-world scenario where a PDB with minAvailable set too high prevented a node from being drained, causing a cascade of failed evictions and stuck deployments. The conversation covers the difference between minAvailable and maxUnavailable, how PDBs interact with voluntary disruptions and node drains, and practical tuning strategies to avoid over-constraining your cluster. They also discuss monitoring PDB status and using tools like Descheduler and PDB Radar to prevent stalls. If you've ever watched a node drain hang indefinitely, this episode gives you the mental model and the concrete fixes you need. Tune in for a focused, jargon-free breakdown of a classic Kubernetes failure mode.

#Kubernetes #PodDisruptionBudgets #NodeDrain #DevOps #CI #CD #SRE #CloudNative #ContainerOrchestration #ClusterOperations #TechOps #InfrastructureAsCode #FexingoBusiness #BusinessPodcast #TechnologyPodcast #DevOpsDaily #K8s #KubernetesTroubleshooting

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily, Lucas and Luna dig into a nasty failure mode that has bitten many production clusters: the kubelet's image garbage collector deleting images that running containers still need. They walk through a real scenario where a node with a small disk runs out of space, the kubelet's image GC kicks in and removes the image for a container that is actively running, and the container gets killed with a 'No such image' error. They explain the difference between image garbage collection and container garbage collection, the default thresholds (HighThresholdPercent and LowThresholdPercent), and why the kubelet's behavior can be surprising when you have a container that is running but not recently started. They discuss how to check disk usage, set proper thresholds, and use image pull policies and pod eviction budgets to avoid this silent killer. Along the way, they share war stories and practical tips for operators, and they cover how to monitor and alert on disk pressure before it becomes a full-blown incident. Tune in for a focused, actionable look at a Kubernetes gotcha that can take down your workloads when you least expect it.

#Kubernetes #Kubelet #ImageGarbageCollection #ContainerGarbageCollection #DevOps #SRE #NodeOperations #DiskPressure #ImagePullPolicy #K8sTroubleshooting #ProductionFailure #TechPodcast #FexingoBusiness #BusinessPodcast #Technology #ITOperations #CloudNative #KubernetesTroubleshooting

Keep every episode free: buymeacoffee.com/fexingo

On this milestone 150th episode, Lucas and Luna dig into a Kubernetes failure mode that quietly grinds cluster upgrades to a halt: Pod Disruption Budgets. Using the story of a mid-sized fintech that got stuck mid-drain for six hours, they explain how PDBs interact with node drains, voluntary disruptions, and replica counts, and why your 'minAvailable: 2' can block a rolling upgrade even when nothing looks wrong. They walk through the warning signs, the kubectl commands that reveal the blockage, and practical fixes like ephemeral burst capacity and drain timeouts. Plus, a short note on why this show stays ad-free and how listener support keeps it that way.

#Kubernetes #PodDisruptionBudgets #NodeDrain #DevOps #SRE #CloudNative #ClusterUpgrades #Kubectl #Fintech #Infrastructure #ReliabilityEngineering #TechOps #SoftwareOperations #FexingoBusiness #BusinessPodcast #TechPodcast #DevOpsDaily #K8sFailureModes

Keep every episode free: buymeacoffee.com/fexingo