DevOps Daily with Fexingo: CI/CD, Kubernetes, and Modern Software Operations

In this episode of DevOps Daily, Lucas and Luna dive into a hidden performance killer in Kubernetes service meshes: the mutual TLS handshake that happens every time a pod starts up. They explain how Istio's mTLS setup creates a cascade of blocking calls that can delay pod readiness by 10 to 30 seconds in large clusters, using a real-world case from a fintech company running 2,000 microservices. Lucas breaks down the mechanics of the Istio proxy's secret discovery service, the impact of certificate rotation, and why a simple startup probe tuning reduced cold-start failures by 40 percent. Luna shares a counterexample from an e-commerce platform that avoided the issue entirely by using mTLS striping on a per-namespace basis. They also discuss practical mitigations like pre-heated sidecar caches and readiness probe delays. The episode closes with a brief, sincere note about how listener support via buy me a coffee dot com slash fexingo funds the show's ad-free production.

#Kubernetes #ServiceMesh #Istio #mTLS #PodStartup #SiteReliabilityEngineering #DevOps #CloudNative #ContainerOrchestration #Fintech #Microservices #SidecarProxy #SecretDiscoveryService #CertificateRotation #Latency #Performance #FexingoBusiness #BusinessPodcast

Keep every episode free: buymeacoffee.com/fexingo

Kubernetes 1.24 introduced the ability to expand PersistentVolumeClaims for StatefulSets, but the feature has a dangerous edge case. When a StatefulSet’s storage template is updated, the controller can trigger concurrent resize operations across multiple replicas, overwhelming the underlying storage backend and causing node disk failures. In this episode, Lucas and Luna walk through a real incident at a mid-sized SaaS company where a simple PVC size bump from 50GB to 100GB caused a cascading failure across three nodes. They explain the mechanics of the resize API, the race condition in generic ephemeral volumes, and the mitigation strategies—including pod-level PDBs and phased rollouts. DevOps engineers currently running StatefulSets with expanded PVCs will walk away with a concrete checklist to avoid this silent cluster-killer.

#Kubernetes #StatefulSet #PVCResize #NodeDiskFailure #PersistentVolumeClaim #DevOps #CloudNative #InfrastructureFailure #RaceCondition #ETCD #Kubelet #StorageBackend #Persistence #CascadingFailure #Technology #FexingoBusiness #BusinessPodcast #ProductionIncident

Keep every episode free: buymeacoffee.com/fexingo

Episode 87 of DevOps Daily with Fexingo dives into a subtle Kubernetes scheduling pitfall: topology spread constraints that unintentionally create node hotspots. Lucas and Luna walk through a real scenario where evenly-distributed pod replicas on paper led to imbalanced cluster utilization in practice. They explain how the `maxSkew` parameter interacts with node labels, and why a 1-milli-core resource request difference can cascade into scheduling failures. The hosts also share a debugging story from a production deployment that misconfigured spread constraints with multiple topology keys, causing 30% of pending pods to timeout. If you've ever tuned pod topology and got unexpected results, this episode surfaces the silent skew that breaks balance.

#Kubernetes #Scheduling #TopologySpreadConstraints #MaxSkew #PodScheduling #ClusterBalancing #DevOps #K8s #NodeLabels #PodDistribution #Scheduler #ResourceRequests #ProductionDebugging #KubernetesTips #CloudNative #Technology #FexingoBusiness #BusinessPodcast

Keep every episode free: buymeacoffee.com/fexingo

Episode 86 of DevOps Daily with Fexingo dives into a subtle but painful Kubernetes trap: Custom Resource Definition (CRD) versioning. When you update a CRD's schema to add a new field or change a validation rule, existing Custom Resources stored in etcd may not automatically convert to the new version—and controllers that watch the old version can crash or miss updates. We walk through a real incident at a mid-sized SaaS company where a rollback attempt actually made things worse, because the old controller couldn't parse the new CRD storage version. You'll learn why the `storage` flag matters, how conversion webhooks can save you, and one simple check to add to your CI/CD pipeline before deploying CRD changes.

#Kubernetes #CustomResourceDefinitions #CRDVersioning #ControllerUpgrades #etcd #KubernetesOperators #ConversionWebhooks #DevOps #SiteReliabilityEngineering #KubernetesRollback #CI/CD #CloudNative #KubernetesStorage #ControllerManager #APIExtensions #KubernetesTroubleshooting #FexingoBusiness #BusinessPodcast

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily with Fexingo, Lucas and Luna dive into a subtle but costly Kubernetes performance issue: how verbose audit logging can silently degrade etcd performance, leading to increased latency, timeouts, and even cluster instability. They walk through a real-world case from a mid-sized fintech that saw API response times jump from 5ms to over 200ms after enabling audit logs without tuning. You'll learn about the audit log pipeline, how etcd handles writes, and three practical mitigations: adjusting audit policy verbosity, using webhook batching, and offloading logs via sidecar. No theory—just actionable advice for anyone running Kubernetes in production.

#Kubernetes #AuditLogging #etcd #Performance #CloudNative #SRE #DevOps #ProductionIncident #Fintech #K8s #Logging #Latency #ClusterStability #Monitoring #Tuning #Technology #FexingoBusiness #BusinessPodcast

Keep every episode free: buymeacoffee.com/fexingo

Episode 84 of DevOps Daily with Fexingo. Lucas and Luna investigate a subtle but dangerous bug in Kubernetes volume snapshot workflows. They break down how the CSI snapshot controller, combined with certain storage backends, can lead to silent data corruption when snapshots are taken during active I/O. Lucas walks through the sequence of events: the VolumeSnapshot CRD, the VolumeSnapshotContent object, and the timing gap where the CSI driver creates a snapshot before the filesystem is quiesced. Luna challenges whether this is a Kubernetes issue or a storage vendor problem, and they discuss real-world mitigations like fs-freeze hooks and snapshot consistency groups. The episode ends with practical advice for cluster operators running stateful workloads.

#Kubernetes #VolumeSnapshots #DataCorruption #CSIStorage #StatefulWorkloads #CloudNative #DevOps #Infrastructure #StorageBackend #Snapshots #K8sStorage #ContainerStorage #Backup #DisasterRecovery #Technology #DevOpsDaily #FexingoBusiness #BusinessPodcast

Keep every episode free: buymeacoffee.com/fexingo

Episode 83 of DevOps Daily with Fexingo dives into a subtle but costly Kubernetes issue: Custom Resource Definitions (CRDs) and the controllers built to manage them can silently leak memory over weeks of operation. Lucas and Luna walk through a real incident at a mid-sized fintech company where in-memory caches of CRD objects grew unbounded, leading to OOM-killed controllers and cascading failures. They explain why the problem stems from the way client-go's informer cache interacts with custom resources under high-churn workloads, and what operators can do to catch it before it hits production. If you manage a Kubernetes cluster with custom resources, this episode will change how you monitor controller memory.

#Kubernetes #CRD #CustomResourceDefinitions #MemoryLeak #Controller #ClientGo #InformerCache #GoProgramming #OOMKill #Fintech #ProductionIncident #KubernetesOperator #CloudNative #SiteReliabilityEngineering #DevOps #Technology #FexingoBusiness #BusinessPodcast

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily, Lucas and Luna dive into a subtle Kubernetes scheduling trap: how PodDisruptionBudgets, meant to ensure high availability during voluntary disruptions, can inadvertently cause rollout stalls in production. Using a real-world example from an e-commerce platform handling Black Friday traffic, they explain the interplay between PDBs, replica counts, and node drains. Lucas walks through the scenario where a PDB with 'maxUnavailable: 1' combined with a two-replica deployment and a rolling update leads to a complete freeze. They discuss the root cause, how the kube-scheduler and kube-controller-manager interact with PDBs, and practical mitigations like using 'minAvailable' over 'maxUnavailable', setting appropriate replica counts, and monitoring with custom alerts. Listeners walk away with a concrete debugging approach and a better understanding of why their rollouts sometimes stall for no apparent reason.

#Kubernetes #PodDisruptionBudget #DevOps #Scheduling #RolloutStall #HighAvailability #KubeScheduler #NodeDrain #Ecommerce #BlackFriday #MaxUnavailable #MinAvailable #RollingUpdate #ReplicaCount #KubernetesTroubleshooting #CloudNative #SRE #FexingoBusiness

Keep every episode free: buymeacoffee.com/fexingo

In Episode 81 of DevOps Daily with Fexingo, Lucas and Luna dive into a specific blind spot in Kubernetes observability: the Node Problem Detector (NPD). They explain how NPD's default configuration relies on static patterns and often misses subtle hardware and kernel-level issues like memory corruption, NIC firmware bugs, and ACPI errors that can silently degrade node health. Lucas walks through a real-world case where a data center experienced mysterious pod evictions traced back to a faulty CPU cache line that NPD never flagged. The hosts discuss alternatives like using eBPF-based probes and integrating with node-problem-detector's custom plugin mechanism to catch these silent failures. They also touch on the economics of running open-source alerting stacks without ads. If you manage Kubernetes clusters in production, this episode gives one concrete gap to check in your monitoring setup.

#NodeProblemDetector #Kubernetes #Observability #eBPF #SilentFailures #DevOps #CloudNative #SRE #LinuxKernel #HardwareMonitoring #PodEvictions #ACPIErrors #NodeHealth #OpenSourceFunding #BuyMeACoffee #FexingoBusiness #BusinessPodcast #Technology

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily with Fexingo, Lucas and Luna dive into a subtle but destructive Kubernetes behavior: how ResourceQuotas can silently evict pods when namespace limits are reached, even when the cluster has ample capacity. They walk through a real incident at a mid-sized e-commerce company where a single namespace's quota misconfiguration caused cascading evictions across 12 microservices during a flash sale. Lucas explains the mechanism—how the kube-apiserver rejects pod creations when quota is exceeded, and how the scheduler's lack of awareness leads to orphaned pods. Luna raises the issue of observability gaps, noting that standard dashboards often miss quota-related denials. They discuss mitigation strategies: setting explicit deny messages, monitoring quota metrics via Prometheus, and using admission webhooks for early warnings. The episode delivers a concrete lesson for any team running multi-tenant clusters. A brief donation segment highlights listener support for the ad-free show.

#Kubernetes #ResourceQuotas #PodEvictions #DevOps #CloudNative #K8sTroubleshooting #ClusterManagement #AdmissionControl #NamespaceQuotas #Observability #Prometheus #IncidentResponse #Ecommerce #FexingoBusiness #BusinessPodcast #Technology #DevOpsDaily #Infrastructure

Keep every episode free: buymeacoffee.com/fexingo