In this episode of DevOps Daily, Lucas and Luna dig into a nasty failure mode that has bitten many production clusters: the kubelet's image garbage collector deleting images that running containers still need. They walk through a real scenario where a node with a small disk runs out of space, the kubelet's image GC kicks in and removes the image for a container that is actively running, and the container gets killed with a 'No such image' error. They explain the difference between image garbage collection and container garbage collection, the default thresholds (HighThresholdPercent and LowThresholdPercent), and why the kubelet's behavior can be surprising when you have a container that is running but not recently started. They discuss how to check disk usage, set proper thresholds, and use image pull policies and pod eviction budgets to avoid this silent killer. Along the way, they share war stories and practical tips for operators, and they cover how to monitor and alert on disk pressure before it becomes a full-blown incident. Tune in for a focused, actionable look at a Kubernetes gotcha that can take down your workloads when you least expect it.

#Kubernetes #Kubelet #ImageGarbageCollection #ContainerGarbageCollection #DevOps #SRE #NodeOperations #DiskPressure #ImagePullPolicy #K8sTroubleshooting #ProductionFailure #TechPodcast #FexingoBusiness #BusinessPodcast #Technology #ITOperations #CloudNative #KubernetesTroubleshooting

Keep every episode free: buymeacoffee.com/fexingo

On this milestone 150th episode, Lucas and Luna dig into a Kubernetes failure mode that quietly grinds cluster upgrades to a halt: Pod Disruption Budgets. Using the story of a mid-sized fintech that got stuck mid-drain for six hours, they explain how PDBs interact with node drains, voluntary disruptions, and replica counts, and why your 'minAvailable: 2' can block a rolling upgrade even when nothing looks wrong. They walk through the warning signs, the kubectl commands that reveal the blockage, and practical fixes like ephemeral burst capacity and drain timeouts. Plus, a short note on why this show stays ad-free and how listener support keeps it that way.

#Kubernetes #PodDisruptionBudgets #NodeDrain #DevOps #SRE #CloudNative #ClusterUpgrades #Kubectl #Fintech #Infrastructure #ReliabilityEngineering #TechOps #SoftwareOperations #FexingoBusiness #BusinessPodcast #TechPodcast #DevOpsDaily #K8sFailureModes

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily, Lucas and Luna dig into a surprisingly common but often overlooked failure mode in Kubernetes: container runtime interface timeouts. When the kubelet's connection to the container runtime — typically containerd or CRI-O — goes stale or hangs, node drains can stall, pods get stuck in Terminating, and your carefully planned maintenance window turns into a fire drill. Lucas walks through a real-world scenario where a single slow CRI call blocked an entire node drain, leading to failed deployments and user-facing outages. They discuss why timeouts happen, how to spot them in kubelet logs, and what you can do to prevent them — from tuning the kubelet's node-status-update-frequency to setting proper runtime-level timeouts. If you've ever seen a pod stuck in Terminating with no obvious cause, this episode will give you the diagnostic path and the fix.

#Kubernetes #CRITimeout #NodeDrain #ContainerRuntime #Containerd #DevOps #SiteReliabilityEngineering #Kubelet #PodTermination #CloudNative #InfrastructureAsCode #TechPodcast #BusinessPodcast #FexingoBusiness #BusinessPodcast #TechOps #CI #CD

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily, hosts Lucas and Luna dive into the hidden dangers of Kubernetes Topology Spread Constraints. They explain how misconfigured spread constraints can prevent pods from scheduling, cause resource fragmentation, and trigger unexpected failovers. Using a real-world example of a three-node cluster with a rollout gone wrong, they walk through the mechanics of how these constraints interact with node affinity and taints, and why the scheduler sometimes makes counterintuitive decisions. They also share practical tips for auditing spread constraints and avoiding common pitfalls. If you manage Kubernetes clusters, this episode will save you from a late-night debugging session.

#Kubernetes #TopologySpreadConstraints #PodScheduling #DevOps #ContainerOrchestration #ClusterManagement #Scheduler #Rollout #HighAvailability #InfrastructureAsCode #TechOps #CloudNative #PlatformEngineering #FexingoBusiness #BusinessPodcast #Technology #SoftwareEngineering #SRE

Keep every episode free: buymeacoffee.com/fexingo

In this episode, Lucas and Luna dig into a nasty failure mode that shows up when ArgoCD's sync phases and hooks are misused: a single application rollout that triggers a cascade of errors, from failed health checks to stuck syncs. They walk through a real-world scenario from a mid-size fintech's production cluster, where a seemingly straightforward update to a payments service brought down dependent services because the sync wave ordering clashed with readiness gates. They explain how ArgoCD's sync waves and resource hooks work, why the default 'sync' phase is often misunderstood, and how a team can end up with a deployment that looks successful in the UI but is actually broken underneath. They share practical guardrails: using sync waves deliberately, matching them with readiness probes, and always testing the sync order in a staging cluster before trusting the production pipeline. If you've ever wondered why your ArgoCD app says 'Synced' but your users are seeing errors, this episode is for you.

#ArgoCD #Kubernetes #GitOps #CICD #DevOps #SyncWaves #SyncHooks #DeploymentFailures #ReadinessProbes #Fintech #ProductionOutages #ContinuousDeployment #InfrastructureAsCode #SoftwareReliability #SiteReliabilityEngineering #CloudNative #FexingoBusiness #BusinessPodcast

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily with Fexingo, hosts Lucas and Luna uncover a quiet but dangerous gap in Kubernetes security: running Pod Security Admission in audit mode can let risky workloads slip into production without a single warning. They trace a real scenario where a team thought they were protected by PSA audit logs, but misconfigured exclusions and permissive policies meant no alert ever fired. Drawing on the Kubernetes 1.23 introduction of Pod Security Standards, Lucas explains how audit mode differs from enforce mode, why relying on logs alone creates blind spots, and how a simple validation check—like a pre-deployment policy scan—can save your cluster from a nasty surprise. Perfect for platform engineers and SREs who think they've got Pod Security covered. Tune in to learn one concrete takeaway: audit mode is a starting point, not a safety net.

#Kubernetes #PodSecurityAdmission #PodSecurityStandards #AuditMode #DevOps #KubernetesSecurity #PodSecurity #PlatformEngineering #CloudNative #KubernetesTips #SecurityBestPractices #SRE #CI/CD #TechPodcast #FexingoBusiness #BusinessPodcast #DevOpsDaily #KubernetesAdmission

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily, Lucas and Luna dig into a silent data-loss hazard in Kubernetes: the PersistentVolume reclaim policy. They walk through a real incident where a Recycle policy deleted a production database's data during a routine Helm upgrade, and explain the differences between Retain, Recycle, and Delete. They cover why the default Delete policy can be dangerous for stateful workloads, how to audit your cluster for risky PVs, and best practices for protecting persistent data with Retain or modern CSI snapshots. If you run stateful applications on Kubernetes, this episode gives you a concrete checklist to avoid an irreversible mistake.

#Kubernetes #PersistentVolume #DataLoss #DevOps #StatefulWorkloads #Helm #CSI #Storage #RetainPolicy #RecyclePolicy #Tech #Business #FexingoBusiness #BusinessPodcast #CloudNative #Infrastructure #SiteReliability #DataProtection

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily, Lucas and Luna dig into a surprisingly common Kubernetes failure: StatefulSets that keep rolling out old images because the controller's ordering logic interacts badly with image pull policies and rolling update strategies. Using a concrete example from a payment-processing service at a European fintech, they explain why 'Always' pull policy can mask a stale image, why the ordinal index (pod-0, pod-1...) creates a false sense of progress, and how a single misconfigured update strategy can leave pods running old code for weeks. They walk through the exact kubectl commands to verify what image each pod is actually running and how to fix the rollout without deleting the StatefulSet. Along the way, they touch on the difference between onDelete and RollingUpdate, the role of revision history, and why you should never trust 'kubectl rollout status' alone. If you've ever scratched your head wondering why your StatefulSet isn't updating, this episode gives you a mental model and a debugging path.

#Kubernetes #StatefulSet #DevOps #CI/CD #RollingUpdate #ImagePullPolicy #Kubectl #PodOrdinal #Fintech #DeploymentStrategies #ContainerOrchestration #TechOps #SoftwareDeployment #CloudNative #K8sTroubleshooting #FexingoBusiness #BusinessPodcast #TechPodcast

Keep every episode free: buymeacoffee.com/fexingo

In this episode, Lucas and Luna dig into a production incident where a service mesh's mutual TLS certificates expired overnight, silently killing traffic between microservices. They walk through the failure mode: certificates that were set to rotate every 24 hours but got stuck because of a clock skew issue on one of the worker nodes. They explain how Kubernetes CertificateSigningRequest objects work, why the rotation loop can fail silently, and what happens when the mesh's control plane doesn't fail open. They also discuss practical mitigations, like monitoring certificate expiry as a first-class signal, setting up alerting on rotation failures, and using short-lived certificates with a well-tested renewal path. The episode includes a concrete checklist for anyone running a service mesh in production.

#Kubernetes #ServiceMesh #mTLS #CertificateRotation #DevOps #SiteReliability #ProductionIncident #Microservices #Istio #Linkerd #KubernetesCertificates #TLS #ClockSkew #Monitoring #Alerting #FexingoBusiness #BusinessPodcast #TechOps

Keep every episode free: buymeacoffee.com/fexingo

In this episode, Lucas and Luna dig into a surprisingly common Kubernetes failure mode: when Pod PriorityClasses and preemption interact badly with cluster autoscaling, nodes get drained, workloads get killed, and the cluster autoscaler ends up fighting itself. They walk through a real-world scenario where a high-priority batch job preempts a low-priority web service, the node autoscaler scales down the now-idle node, and the preempted pods can't reschedule because the cluster is too small — creating a deadlock. Lucas explains how preemption is designed to work, why the autoscaler's node utilization threshold can trigger a scale-down right after preemption, and what signals you can look for in events and metrics to catch this before it bites. They also discuss practical mitigations: using PodDisruptionBudgets with preemption, tuning the autoscaler's scale-down utilization threshold, and considering whether preemption is even the right tool for your workload. If you run a cluster with mixed priorities and autoscaling, this episode gives you a concrete failure pattern to recognise and avoid.

#Kubernetes #PodPriority #Preemption #ClusterAutoscaling #DevOps #CloudNative #SRE #PodDisruptionBudget #KubernetesOperators #InfrastructureAsCode #Tech #FexingoBusiness #BusinessPodcast #DevOpsDaily #KubernetesTroubleshooting #Autoscaling #PriorityClass #KubernetesBestPractices

Keep every episode free: buymeacoffee.com/fexingo