OpenObservability Talks

Is Kubernetes Monitoring Flawed? - OpenObservability Talks S3E08


Listen Later

A 3-node Kubernetes cluster with Prometheus will ship around 40k active series by default! Do we really need all that data? The current state of Kubernetes open source monitoring is in need of improvement. High churn rate of pod metrics, proliferation of metrics with low usage, and configuration complexity are some of the issues that need to be addressed.

I discussed this topic with Aliaksandr Valialkin, CTO at VictoriaMetrics and creator of the open source project. We discussed the common problems, as well as directions and best practices to overcome some of these complexities as individuals and as a community. We also discussed VictoriaMetrics open source project and how it addresses some of these challenges.

Aliaksandr a Golang engineer, who likes writing simple and performant code and creating easy-to-use programs. Sometimes these hard-to-match requirements work together, like in the VictoriaMetrics case.

The episode was live-streamed on 24 January 2023 and the video is available at https://www.youtube.com/live/Z-58C8HFGb8

OpenObservability Talks episodes are released monthly, on the last Thursday of each month and are available for listening on your favorite podcast app and on YouTube.

We live-stream the episodes on Twitch and YouTube Live - tune in to see us live, and chime in with your comments and questions on the live chat.
https://www.twitch.tv/openobservability
https://www.youtube.com/@openobservabilitytalks

Have you got an interesting topic you'd like to share in an episode? Reach out to us and submit your proposal at https://openobservability.io/

Show Notes:

  • monitoring microservice system, app and communications
  • high churn rate for pod metrics
  • Kubernetes produces too many metrics by defaults, most of which are unused
  • recommended listing of metrics
  • removing unused metric labels to reduce cardinality
  • Prometheus native (exponential buckets) historgrams
  • Configuration complexity with multiple deployments
  • OpenTelemetry and OpenMetrics open specifications
  • collecting system metrics and application metrics uniformly
  • VictoriaMetrics essentials
  • VictoriaMetrics extensions beyond Prometheus
  • a full stack monitoring collection, analysis and alerting
  • how to join the VictoriaMetrics community
  • industry update: 2023 cloud native predictions post by CNCF CTO
  • Resources:

    • Why Prometheus cannot query remote storage in an expected way via remote_read protocol - https://github.com/prometheus/prometheus/issues/4456 
    • VictoriaMetrics scaling to 100 million metrics per second https://www.youtube.com/watch?v=xfed9_Q0_qU
    • https://victoriametrics.com/
    • https://github.com/VictoriaMetrics/VictoriaMetrics
    • https://docs.victoriametrics.com/#community-and-contributions
    • Socials:

      • Twitter: https://twitter.com/OpenObserv
      • Twitch: https://www.twitch.tv/openobservability
      • YouTube: https://www.youtube.com/@openobservabilitytalks
      • Dotan Horovits 

        Twitter: https://twitter.com/horovits  
        LinkedIn: https://www.linkedin.com/in/horovits/  


        Aliaksandr Valialkin

        Twitter: https://twitter.com/valyala

        LinkedIn: https://www.linkedin.com/in/valyala/

        ...more
        View all episodesView all episodes
        Download on the App Store

        OpenObservability TalksBy Dotan Horovits

        • 5
        • 5
        • 5
        • 5
        • 5

        5

        3 ratings


        More shows like OpenObservability Talks

        View all
        The Changelog: Software Development, Open Source by Changelog Media

        The Changelog: Software Development, Open Source

        289 Listeners

        Software Engineering Daily by Software Engineering Daily

        Software Engineering Daily

        626 Listeners

        The Cloudcast by Massive Studios

        The Cloudcast

        153 Listeners

        Affaires sensibles by France Inter

        Affaires sensibles

        298 Listeners

        The Daily by The New York Times

        The Daily

        112,416 Listeners

        Syntax - Tasty Web Development Treats by Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers

        Syntax - Tasty Web Development Treats

        988 Listeners

        Kubernetes Podcast from Google by Abdel Sghiouar, Kaslin Fields

        Kubernetes Podcast from Google

        182 Listeners

        AWS Podcast by Amazon Web Services

        AWS Podcast

        202 Listeners

        DevOps Paradox by Darin Pope & Viktor Farcic

        DevOps Paradox

        25 Listeners

        AWS Morning Brief by Corey Quinn

        AWS Morning Brief

        79 Listeners

        Dwarkesh Podcast by Dwarkesh Patel

        Dwarkesh Podcast

        501 Listeners

        Developer Voices by Kris Jenkins

        Developer Voices

        30 Listeners

        Rust in Production by Matthias Endler

        Rust in Production

        26 Listeners

        KubeFM by KubeFM

        KubeFM

        2 Listeners

        The Pragmatic Engineer by Gergely Orosz

        The Pragmatic Engineer

        64 Listeners