Data on Kubernetes Community

Almost all applications have some kind of state. Some data processing apps and databases have huge amounts of state. How do we navigate a cloud-based world of containers where stateless and functions-as-a-service is all the rage? As a long-time architect, designer, and developer of very stateful apps (databases and data processing apps), I’d like to take you on a journey through the modern cloud world and Kubernetes, offering helpful design patterns, considerations, tips, and where things are going. How is Kubernetes shaking up stateful app design?  - What kind of state is there, and what are some important characteristics? - Kubernetes, containers, and the stateless paradigm (pushing state into DBs) - Where state lives and the persistence characteristics - Stateless vs serverless - why stateless is not really stateless, but server less really is - Improving on stateless paradigm using local state pattern - Logs and event streaming for reasoning about state and failure recovery - The case for local disks: ML, Databases, etc. - Kubernetes and the Persistent Volume/StatefulSets - Leveraging Kubernetes PVs as a basis for building distributed data systems - Mapping the solution space

Evan has been a distributed systems / data / software engineer for twenty years. He led a team developing FiloDB, an open source (github.com/filodb/FiloDB) distributed time series database that can process a million records per second PER NODE and simultaneously answer a large number of concurrent queries per second. He has architected, developed, and productionized large scale data and telemetry systems at companies including Apple, and loves solving the most challenging technical problems at both large and small scales, from advanced custom data structures to distributed coordination. He is an expert in bleeding edge #jvm #java #scala and #rust performance. Current interests include Rust and columnar compression. He has led the design and implementation of multiple big data platforms based on Apache Storm, Spark, Kafka, Cassandra, and Scala/Akka. He has been an active contributor to the Apache Spark project, and a two-time Datastax Cassandra MVP.

Chaos Engineering is revolutionizing testing means and doing it the cloud-native way is the best way in today's rapidly changing world with a huge shift in the paradigm of Kubernetes resiliency. Karthik S, one of the maintainers for LitmusChaos would be introducing how to carry out Chaos Engineering, the cloud-native way. Further, he will touch upon how Chaos Engineering is carried out in Cloud-Native Databases with LitmusChaos. He will also touch upon observability considerations for chaos engineering and what hooks Litmus provides for the same.

Karthik Satchitanand is one of the maintainers of the CNCF sandbox project LitmusChaos. He is passionate about all things Kubernetes, and is generally interested in DevOps, storage performance/benchmarking & chaos engineering.

A fireside chat with Jérôme Petazzoni in which we will get to know him up close and personal, ask him about how his personal music projects influence his professional work, and answer questions from the audience.

Jérôme was part of the team that built, scaled, and operated the dotCloud PAAS, before it became Docker. He worked seven years at the famous container company, wearing various hats. When he's not busy with computers, he collects musical instruments. He can arguably play the theme of Zelda on a dozen of them.

In this talk we will discuss what are the best patterns to create stateful applications on top of Kubernetes. This will include application layer caching, embeddable database as well as leveraging kubernetes objects to store and sync state across multiple replicas.

Prashant is passionate about democratizing best AppOps practices for Kubernetes and has started Devtron and open source platform for AppOps on top of Kubernetes. Recently open sourced Devtron has been instrumental in adoption of Kubernetes in some of the largest startups in India.

ENG: Cloud&Heat has always focused on providing energy-efficient data centers. In the last 8 years, we have developed an innovative water cooling technology for servers, converting waste heat into a valuable asset. By doing so, we have already greatly improved the energy efficiency of individual data centers. However, this isn’t enough. To globally maximize the efficiency of distributed data center infrastructures, this talk presents Krake. Krake is an orchestration software for compute-intensive jobs. It improves the global cost and energy efficiency of infrastructures by balancing the load between data centers. Krake evaluates and selects the most efficient site to run jobs based on certain metrics, such as energy availability, heat demand, and latency. It also reacts to changes in the system by migrating jobs. In other words, it ensures a job is run in the most energy- and/or cost-efficient way at any given time.

ES: Juan A. Fraire es investigador asistente en CONICET en Argentina, profesor asociado en la Universidad de Saarland en Alemania y arquitecto de soluciones cloud en Cloud&Heat en Alemania. Su principal interés gira en torno a las aplicaciones en red y distribuidas habilitadas por técnicas informáticas de vanguardia. Además de aplicar esto al dominio del cloud computing en tierra, Juan lidera un esfuerzo internacional para llevar la visión al dominio del espacio. De hecho, Juan es el fundador y presidente del STINT Workshop desde 2014, ha sido coautor de más de 55 artículos publicados en revistas internacionales y conferencias líderes, y ha construído una visión novedosa de redes en su libro “Delay-Tolerant Satellite Network ” en coautoría con colegas de la NASA. 

EN: Juan A. Fraire is an assistant researcher at CONICET in Argentina, an associate professor at Saarland University in Germany, and cloud solutions architect at Cloud&Heat in Germany. His main interest revolves around networking and distributed applications enabled by state-of-the-art informatics techniques. Besides applying this to the cloud domain on ground, Juan leads an international effort in pushing the vision into the space domain. Indeed, Juan is the founder and chair of the annual Space-Terrestrial Internetworking Workshop (STINT) since 2014, has co-authored more than 55 papers published in international journals and leading conferences, and has imprinted a novel networking vision in his “Delay-Tolerant Satellite Network” book co-authored with colleagues from NASA.

Global load balancing, commonly referred to as GSLB (Global Server Load Balancing) solutions, have typically been the domain of proprietary network software and hardware vendors and installed and managed by siloed network teams. k8gb is a completely open source, cloud native, global load balancing solution for Kubernetes. k8gb focuses on load balancing traffic across geographically dispersed Kubernetes clusters using multiple load balancing strategies to meet requirements such as region failover for high availability. Global load balancing for any Kubernetes Service can now be enabled and managed by any operations or development teams in the same Kubernetes native way as any other custom resource. The talk will cover both technical and business aspects of k8gb creation including ongoing adoption within the huge scale organization.

Yury is an experienced software engineer with a strong focus on Linux, Kubernetes and software quality. He is passionate about open source and contributes to several upstream projects on a regular basis. Currently, he is focused on developing open-source distributed systems in cloud-native environments.

Our learnings from Kubernetes

Hundreds of data teams have migrated to the ELT pattern in recent years, leveraging SaaS tools like Stitch or FiveTran to reliably load data into their infrastructure. These SaaS offerings are outstanding and can accelerate your time to production significantly. However, many teams prefer to roll their own tools. One solution in these cases is to deploy singer.io taps and targets — Python scripts that can perform data replication between arbitrary sources and destinations. The Singer specification is the foundation for the popular Stitch SaaS, and it is also leveraged by a number of independent consultants and data projects. Singer pipelines are highly modular. You can pipe any tap to any target to build a data pipeline that fits your needs, making them a good fit for containerized workflows. This article walks through the workflow at a high level and provides some example code to get up and running with some shared templates. I also drill into reasons for choosing the Argo approach over other orchestration tools like Airflow or Dagster, and the implications from a team perspective.

Stephen Bailey is Director of Growth Analytics at Immuta, where he strives to implement privacy best practices while delivering business value from data. He loves to teach and learn, on just about any subject. He holds a PhD in educational cognitive neuroscience from Vanderbilt and enjoys reading philosophy

Junior pursuing Computer Science & Engineering. Co-founder at Code for Cause. CNCF Intern 2020. MLH Coach. Google Summer of Code Mentor. YouTuber. Gold Microsoft Learn Student Ambassador.

DevOps is like a camera. We focus on what's important, we capture the good times, we develop from the negatives, and if things don't work out, we take another shot. Many teams establishing working best practices for their tools improve their time to deliver and ability to scale. However, the real challenges exist outside of tools and technology and many teams today still have questions about DevOps. So, join this session to learn the fundamentals of shaping a DevOps culture. We'll discuss key attributes around people, process, and technology, likening you and DevOps to pro photographers and cameras.

Tiffany Jachja is a technical evangelist at Harness. She is an advocate for better software delivery, sharing applicable practices, stories, and content around modern technologies. Before joining Harness, Tiffany was a consultant with Red Hat's Consulting practice. There she used her experience to help customers build their software applications living in the cloud.