In this episode of DevOps Daily, Lucas and Luna explore a growing pain in Kubernetes clusters upgraded to version 1.24 and later: default ServiceAccount token expiration. They walk through the surprising moment a listener's CI pipeline stopped deploying, trace the root cause to a one-hour token time-to-live, and explain how to migrate to projected volume mounts with sidecar refresh. With practical before-and-after YAML and a warning about legacy SDKs that still assume permanent tokens, this is a timely survival guide for anyone running modern Kubernetes. Also, Lucas ties in a broader theme of deprecation velocity in cloud-native, and Luna pushes back on whether the change was disruptive by design.

#Kubernetes #DevOps #ServiceAccount #TokenExpiration #CI-CD #CloudNative #Kubernetes124 #Security #Kubelet #ProjectedVolume #Technology #PodIdentity #Automation #SRE #FexingoBusiness #BusinessPodcast #DevOpsDaily #Deprecation

Keep every episode free: buymeacoffee.com/fexingo

etcd compaction is a routine maintenance operation that removes historical key-value versions, but when it runs, it can consume significant IO and CPU, leading to increased latency for API server requests. This episode dives into the mechanics: compaction triggers defragmentation of boltdb pages, causing pauses that can spike request latency from milliseconds to seconds. We discuss a real-world incident where a 500-node cluster saw API latency jump to 8 seconds during peak hours due to compaction misconfiguration. Then we cover mitigation strategies: tuning auto-compaction retention, scheduling defrag during low traffic, monitoring etcd metrics, and setting alerts for compaction duration. Understanding this often-overlooked performance pitfall is crucial for maintaining responsive clusters at scale.

#Kubernetes #etcd #APIServer #Latency #Compaction #Defragmentation #Performance #Tuning #SRE #DevOps #Infrastructure #CloudNative #FexingoBusiness #BusinessPodcast #Technology #Operations #Cluster #Scalability

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily, Lucas and Luna investigate a production outage triggered by Helm upgrade failure after Kubernetes ConfigMaps were set to immutable. They trace the root cause to a Helm chart that marked ConfigMaps immutable, preventing updates without manual deletion. The hosts explain how this feature, intended for performance optimization, can silently break deployments, and share a concrete workaround using a new ConfigMap name per revision. They also discuss how to audit existing deployments for the immutability flag and when it makes sense to use it.

#Kubernetes #Helm #ConfigMap #ImmutableConfigMap #DevOps #CI/CD #DeploymentFailure #HelmUpgrade #KubernetesTroubleshooting #Prometheus #ConfigurationManagement #DevOpsDaily #Fexingo #Technology #KubernetesConfig #HelmChart #Infrastructure #SRE

Keep every episode free: buymeacoffee.com/fexingo

When a Kubernetes cluster went down for 12 minutes last Tuesday, the Node Problem Detector didn't report a thing. Lucas and Luna dig into how this popular add-on can actually mask kernel panics instead of surfacing them. They walk through a real-world scenario: a race condition between the node-problem-detector's event reporting and the API server's watch timeout, causing a false all-clear. Then they explore why the default configuration treats kernel log parsing as informational rather than critical, and how engineers at a major streaming service missed repeated panics for weeks. This episode offers a concrete debugging pattern and three config changes that can prevent a silent cluster meltdown.

#Kubernetes #NodeProblemDetector #KernelPanic #ClusterFailure #DevOps #SiteReliabilityEngineering #ContainerOrchestration #LinuxKernel #EventReporting #WatchTimeout #ClusterDebugging #KubernetesTroubleshooting #Technology #Infrastructure #FexingoBusiness #BusinessPodcast #FexingoTech #CIcd

Keep every episode free: buymeacoffee.com/fexingo

Lucas and Luna dissect a common but overlooked Kubernetes failure mode: ImagePullSecrets that silently expire and cause deployments to land in perpetual ImagePullBackOff. They walk through a real scenario—a team whose ECR token expired overnight, taking down three microservices before anyone noticed—and explain why standard dashboards miss it. They cover automation strategies using Reloader or External Secrets Operator, how to set Prometheus alerts on pull-failure metrics, and why CI/CD pipelines need pre-flight secret validation. A practical episode for anyone running production Kubernetes.

#Kubernetes #DevOps #ImagePullSecrets #DeploymentFailures #CI/CD #SecretRotation #Reloader #ExternalSecretsOperator #Prometheus #ContainerRegistry #ECR #ImagePullBackOff #CloudNative #SiteReliability #KubernetesSecrets #ProductionOps #FexingoBusiness #BusinessPodcast

Keep every episode free: buymeacoffee.com/fexingo

ArgoCD's sync waves are a powerful mechanism for ordering Kubernetes resource deployments, but misconfigurations can lead to cascading failures. In this episode, we break down a real-world scenario where a team's database migration ran after the application pods started, causing a ten-minute outage. We explain how sync waves work, the difference between waves and phases, and common mistakes like missing wave dependencies or using the wrong wave number. We also cover how to test wave ordering in a staging environment and why you should always set a long timeout for wave groups. If you use GitOps in production, this episode will save you from a preventable incident.

#ArgoCD #GitOps #Kubernetes #CI/CD #DevOps #SyncWave #Deployment #InfrastructureAsCode #ContinuousDeployment #CloudNative #Technology #FexingoBusiness #BusinessPodcast #Podcast #DevOpsDaily #K8s #ApplicationDeployment #PostIncidentReview

Keep every episode free: buymeacoffee.com/fexingo

In this episode of DevOps Daily, Lucas and Luna dissect a real incident from a mid-sized e-commerce company: a routine deployment that silently failed because a Kubernetes ResourceQuota for CPU was exhausted. They walk through how ResourceQuotas interact with namespace limits, the cryptic error messages that confuse even experienced operators, and why default quotas often catch teams off guard. Lucas shares a diagnostic trick using kubectl describe quota and explains why Prometheus alerts are essential. The conversation covers the difference between ResourceQuota and LimitRange, how to audit quota usage, and a simple script to alert before hitting limits. Whether you're running multi-tenant clusters or just a few namespaces, this episode will save you from a frustrating afternoon of head-scratching.

#Kubernetes #ResourceQuota #DevOps #CI/CD #CloudNative #ClusterOps #SRE #Troubleshooting #Deployment #SilentFailure #Prometheus #kubectl #Namespace #LimitRange #ContainerOrchestration #K8s #FexingoBusiness #BusinessPodcast

Keep every episode free: buymeacoffee.com/fexingo

Episode 134 of DevOps Daily with Fexingo reveals a hidden risk in containerized CI/CD pipelines: BuildKit cache mounts that silently preserve outdated dependencies. Lucas and Luna unpack a real-world case where a team's npm cache mount kept a vulnerable transitive dependency alive for weeks, bypassing lockfile updates. They explain how cache mount invalidation works, why developers overtrust it, and three practical guardrails to keep builds both fast and fresh. If your pipeline relies on --mount=type=cache, this episode will change how you think about build reproducibility and supply chain security.

#DevOps #CI/CD #Kubernetes #BuildKit #Docker #CacheMounts #DependencyRot #SupplyChainSecurity #ContainerSecurity #BuildCaching #CI_Pipeline #InfrastructureAsCode #DevOpsDaily #FexingoBusiness #BusinessPodcast #Technology #SoftwareOperations #Podcast

Keep every episode free: buymeacoffee.com/fexingo

A single Kubernetes NetworkPolicy rule silently blocked traffic to a fraud detection engine for 48 hours, costing a team thousands in debug time. In this episode, we dissect how NetworkPolicy's default-deny behavior can create invisible connectivity gaps. We explore podSelector, namespaceSelector, and IPBlock gotchas, discuss why policies on one tier cascade to downstream services, and show how to audit policies with kubectl and iptables. We also touch on service mesh layers (Cilium, Istio) that can compound or simplify the problem. If you've ever chased a mysterious timeout, this episode will help you spot the silent black hole before it hits production.

#Kubernetes #NetworkPolicy #DevOps #CloudNative #ContainerNetworking #Microservices #SiteReliabilityEngineering #Tech #FexingoBusiness #BusinessPodcast #DevOpsDaily #K8s #Security #NetworkTroubleshooting #ServiceMesh #CIDR #PodCIDR #Calico

Keep every episode free: buymeacoffee.com/fexingo

In this episode, Lucas and Luna explore a silent traffic-killing bug in Kubernetes Topology Aware Hints. They walk through how the feature, designed to reduce cross-zone network costs, can cause complete traffic blackholes when endpoint slices are missing or misconfigured. Using a real-world example from a three-zone deployment, they explain why kube-proxy drops traffic instead of falling back, and share practical detection and mitigation strategies — including enabling metrics, validating endpoint distribution, and setting appropriate topology keys. If you run multi-zone Kubernetes clusters, this episode will save you from a frustrating outage.

#Kubernetes #TopologyAwareHints #Networking #TrafficBlackhole #KubeProxy #EndpointSlice #CloudNative #DevOps #K8sTroubleshooting #CrossZoneTraffic #SiteReliabilityEngineering #ServiceDiscovery #KubernetesPatterns #TechPodcast #FexingoBusiness #BusinessPodcast #Technology #Infrastructure

Keep every episode free: buymeacoffee.com/fexingo