DevOps Daily with Fexingo: CI/CD, Kubernetes, and Modern Software Operations

In this episode of DevOps Daily, Lucas and Luna dive into a lesser-known Kubernetes behavior: the OOMKill taint. When a node runs out of memory and the kernel kills a pod, Kubernetes applies a taint that can evict all remaining pods on that node, causing a cascade of failures. They walk through a real-world scenario where a single memory-hungry pod triggered node-level taints, leading to widespread pod evictions and a near-total cluster outage. Lucas explains how kube-controller-manager applies the TaintNodeByCondition with the MemoryPressure condition, and why the default taint effect (NoExecute) can be more destructive than expected. They discuss mitigation strategies: setting resource requests and limits conservatively, using PriorityClass to protect critical workloads, and configuring the kubelet's --eviction-hard thresholds. The episode also covers how to detect OOMKill taints early using Prometheus alerts and kubectl describe node. By the end, listeners will understand why OOMKill is not just a pod-level problem but a node-level risk that demands proactive monitoring and resource governance.

#Kubernetes #OOMKill #ClusterReliability #DevOps #CloudNative #NodeTaint #MemoryPressure #PodEviction #kubelet #PriorityClass #Prometheus #ResourceManagement #SRE #ContainerOrchestration #ProductionIncidents #Technology #FexingoBusiness #BusinessPodcast

Keep every episode free: buymeacoffee.com/fexingo

In DevOps Daily Episode 127, Lucas and Luna explore how Kubernetes Pod Security Standards (PSS) can block common supply chain attack vectors. They dissect a real-world scenario where a compromised container image from a public registry tries to escalate privileges via hostPath mounts. Lucas explains the three PSS levels—Privileged, Baseline, and Restricted—and how migrating from the deprecated PodSecurityPolicy to PSS with admission controllers like Kyverno or OPA Gatekeeper can enforce least-privilege policies. They also discuss the attack path where a malicious sidecar container bypasses classic security scans, and how PSS's 'Restricted' profile catches privilege escalation at admission time. The hosts share migration tips, common pitfalls (like blocking legitimate workloads), and why PSS adoption is accelerating as Kubernetes 1.25+ removes PSP. A must-listen for platform engineers and security-conscious DevOps teams.

#Kubernetes #PodSecurityStandards #SupplyChainSecurity #DevSecOps #ContainerSecurity #K8sSecurity #CyberSecurity #CloudNative #AdmissionController #Kyverno #OPAGatekeeper #PodSecurityPolicy #LeastPrivilege #ContainerSupplyChain #TechPodcast #DevOpsDaily #FexingoBusiness #BusinessPodcast

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily with Fexingo, Lucas and Luna dive into a surprisingly common failure mode in Kubernetes: CronJob timezone handling. Since Kubernetes 1.22, the CronJob controller treats schedule strings as UTC by default, but many operators assume it honors the node or container timezone. This leads to jobs running hours early or late, especially during daylight saving transitions. Lucas walks through a real incident where a UK-based team's daily cleanup job ran at 9 PM instead of 2 AM every March and October, causing overnight resource spikes. They discuss the `timeZone` field added in 1.24, how to verify it in older clusters, and why even helm charts often miss this. Luna shares a horror story of a production database backup job that silently missed windows for months. The episode closes with a practical checklist: always set `timeZone`, test across DST boundaries, and monitor actual run times vs expected.

#Kubernetes #CronJob #Timezone #DevOps #CI/CD #CloudNative #SRE #KubernetesCronJob #TimeZoneBug #DaylightSaving #HelmCharts #Kubernetes124 #ProductionIncident #DatabaseBackup #JobScheduling #TechPodcast #FexingoBusiness #BusinessPodcast

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily, Lucas and Luna dive into a notoriously overlooked Kubernetes failure mode—API server overload caused by excessive watch events. They break down how controllers, informers, and custom operators all lean on watch-based architectures, and why a single misconfigured watch can cascade into cluster-wide latency or even outages. Through the lens of a real incident at a mid-sized fintech, they explain what happened when a poorly tuned deployment triggered millions of watch events per minute, flooded etcd, and brought the control plane to its knees. Lucas outlines the specific metrics to monitor (watch event rate, request latency, etcd db size) and the architectural patterns—like using DeltaFIFO, filtering with field selectors, and throttling informer resync intervals—that prevent these meltdowns. Luna brings her own war story about a CI/CD operator that hammered the API server during a Helm release, and the two discuss how to audit your cluster for watch-heavy workloads. By the end, listeners will know exactly how to diagnose and prevent one of the most common yet invisible causes of Kubernetes instability.

#Kubernetes #APIServer #WatchEvents #Etcd #K8sOperators #InformerPattern #ClusterReliability #DevOps #CI/CD #Fintech #SiteReliabilityEngineering #K8sPerformance #ControlPlane #DeltaFIFO #FieldSelectors #ContainerOrchestration #FexingoBusiness #BusinessPodcast

Keep every episode free: buymeacoffee.com/fexingo

Kubernetes Vertical Pod Autoscaler (VPA) is supposed to right-size container resource requests automatically. But in practice, VPA's recommender often suggests CPU and memory values that lead to pod evictions or wasted capacity. Drawing on real incidents from a mid-2026 production cluster running 50 microservices, Lucas and Luna unpack why VPA's default OOM-aware policy can misinterpret short-lived memory spikes, how its recommendation window of 8 days lags behind traffic patterns, and why the 'lower bound' mode sometimes recommends requests below actual usage. They walk through a specific case where VPA recommended 512 MiB of memory for a Go service that actually needed 1.2 GiB under peak load, causing repeated OOMKills. The episode closes with practical mitigations: setting custom OOM-scoring thresholds, using VPA in 'initial' mode for batch workloads, and combining VPA with Horizontal Pod Autoscaler using resource metrics. Listeners come away with a clear mental model of VPA's blind spots and how to compensate for them.

#Kubernetes #VPA #VerticalPodAutoscaler #ResourceRequests #OOMKill #K8sAutoscaling #ContainerSizing #GoService #PodEviction #HPA #ResourceMetrics #ClusterOps #PerformanceTuning #DevOps #Technology #FexingoBusiness #BusinessPodcast #DevOpsDaily

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily, Lucas and Luna dive into a subtle but destructive Kubernetes pitfall: StatefulSet rolling updates that orphan PersistentVolumeClaims when the pod template changes. Using a real-world incident at a mid-sized fintech, they walk through how a simple image update triggered PVC detachment, stalled the rollout, and left the cluster in an inconsistent state. They explain the underlying mechanics — how StatefulSet's pod identity logic interacts with PVC templates, why the 'OnDelete' update strategy buys time but doesn't fix the root cause, and what monitoring signals to watch for. Lucas shares a concrete remediation: pre-binding PVCs via volumeClaimTemplates with stable names, and using a custom operator to handle canary updates safely. Listeners will learn one specific configuration pattern to prevent this silent failure, plus a diagnostic command to check for orphaned PVCs after any StatefulSet change. This is a focused, practical episode for anyone running stateful workloads on Kubernetes.

#Kubernetes #StatefulSet #PersistentVolumeClaim #RollingUpdate #DevOps #CloudNative #StatefulWorkloads #K8sFailure #PodIdentity #VolumeClaimTemplates #OnDeleteStrategy #Fintech #IncidentResponse #KubernetesTroubleshooting #CI/CD #Technology #FexingoBusiness #DevOpsPodcast

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily, Lucas and Luna dive into a subtle but dangerous issue in Kubernetes service meshes: automatic mTLS certificate rotation. When the Istio or Linkerd control plane rotates certificates, badly configured sidecar proxies can restart mid-request, causing connection drops, retry storms, and degraded latency. The hosts walk through a real incident at a mid-size fintech where a 24-hour certificate rotation window triggered cascading failures across 200 microservices. They explain the root cause—a mismatch between the certificate refresh interval and the envoy hot-restart mechanism—and the fix: aligning secrets mount propagation with proxy startup probes. Listeners get a concrete checklist to audit their own mesh configurations before the next rotation hits.

#Kubernetes #ServiceMesh #mTLS #Istio #Linkerd #CertificateRotation #EnvoyProxy #CI/CD #DevOps #SiteReliabilityEngineering #CloudNative #SidecarProxy #PodRestarts #Security #Technology #FexingoBusiness #BusinessPodcast #DevOpsDaily

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily with Fexingo, Lucas and Luna dig into a forgotten Kubernetes feature: PodPreset. Once hailed as a way to inject configuration into pods automatically, PodPreset was deprecated in 2022 and removed in Kubernetes 1.25. But many clusters still carry its legacy — and that legacy causes config drift that operators never see coming. Lucas explains how PodPreset worked, why it was retired in favor of admission webhooks and Operators, and how leftover PodPreset admission controller configurations can silently mutate pod specs. He walks through a real incident where a team spent weeks debugging mysterious container restarts, only to trace the root cause to an old PodPreset definition. Luna pushes on whether the Kubernetes ecosystem moves too fast for production stability. The episode ends with a forward-looking question: as Kubernetes sheds beta APIs and legacy features every release, are operators keeping up with the removal notices buried in release notes? If today's tech conversation gave you something usable, support the show at buy me a coffee dot com slash fexingo.

#Kubernetes #PodPreset #ConfigDrift #DevOps #CloudNative #K8s #AdmissionWebhook #OperatorPattern #KubernetesDeprecation #KubernetesMigration #KubernetesReleaseNotes #KubernetesBetaAPI #KubernetesConfigManagement #CI/CD #Technology #FexingoBusiness #BusinessPodcast #DevOpsDaily

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily with Fexingo, Lucas and Luna dive into a subtle but severe Kubernetes anti-pattern: PriorityClass starvation at the node level. They break down a real-world case where a team set a critical batch job's priority too high, causing it to preempt essential daemonsets and system pods, leading to node instability and failed scheduling. Lucas explains how the Kubernetes scheduler's preemption logic interacts with kubelet's resource allocation, why setting priorities without understanding the full hierarchy can backfire, and the three metrics you should monitor — preemption counts, pod eviction rates, and node pressure conditions — to catch starvation before it takes down your cluster. Luna pushes on whether kubectl describe node gives enough information, and they walk through a practical audit using PriorityClass objects and pod spec priority fields. If you've ever wondered why nodes look healthy but pods won't land, this episode gives you the debugging framework. No fluff, just the edge cases that break production.

#Kubernetes #PriorityClass #PodPreemption #Scheduling #DevOps #CloudNative #NodeStarvation #ClusterReliability #kubelet #BatchJobs #DaemonSet #ResourceManagement #KubernetesScheduler #PodEviction #NodePressure #Technology #FexingoBusiness #BusinessPodcast

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily, Lucas and Luna investigate a deceptively common Kubernetes failure: when ingress controller session affinity (sticky sessions) silently breaks during rolling updates. They walk through a real incident at a mid-size e-commerce platform where users were suddenly logged out mid-session after a routine deployment. Lucas explains the mechanics — how NGINX ingress controller uses Pod IP hashing by default, and how a new replica set flips the hash mapping, redirecting users to pods that have never seen their session data. Luna pushes on whether the fix is a readiness probe tweak, an external session store, or a different ingress controller. They also discuss when to use sticky sessions at all, and why session affinity is often a workaround for a deeper architectural issue. Concrete, specific, and immediately applicable for anyone running Kubernetes in production.

#Kubernetes #IngressController #StickySessions #SessionAffinity #RollingUpdate #NGINXIngress #DevOps #SiteReliabilityEngineering #PodLifecycle #SessionManagement #K8sNetworking #ProductionIncident #ECommerce #LoadBalancing #ReadinessProbe #Technology #FexingoBusiness #BusinessPodcast

Keep every episode free: buymeacoffee.com/fexingo