Data on Kubernetes Community

https://go.dok.community/slack

https://dok.community/ 

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)

The container storage interface (CSI) is a contract between different container orchestrators (Kubernetes, Nomad, etc) and storage plugins. This contract is a set of gRPC services for provisioning, utilizing, and snapshotting storage volumes. In this talk, we will focus on one aspect of the CSI spec: Volume Snapshots.

The talk will cover volume snapshots on Kubernetes CSI, with a deep dive on Kubernetes CSI snapshots. A short demo on the basics of the CSI snapshots on Kubernetes will help show how easy it is to start using CSI snapshots.

After covering the basics on both container schedulers, we’ll deep dive into how backup systems can programmatically interact with the Kubernetes CSI snapshots client. This example will be based on Stork, an open-source backup system that has implemented the CSI spec.

I believe this talk will be beneficial to the community because it covers some storage-related topics that have not been covered much in previous conferences. I’m hoping to bring the unique perspective of a Kubernetes CSI and Nomad CSI contributor, while also covering the CSI plugin maintainer and end-user perspective. With this wide audience, many conference attendees will be able to learn about stateful applications, CSI volume snapshots, and developing CSI drivers for any container scheduler.

Grant Griffiths is a core contributor and reviewer for the Kubernetes CSI and SIG Storage communities working at Portworx. For kubernetes-CSI, he helped bring the external-snapshotter to Beta in Kubernetes 1.17 and GA in Kubernetes 1.20, improved support for secrets, release tooling, and fixed various snapshot controller bugs. At Portworx, he works on control plane components such as the CSI Driver, SDK, CLI, and security. Previously, Grant was a Senior Software Engineer at GE Digital where he built Data Services in Go on k8s.

He has given talks at various community conferences:  

Two talks at GopherCon London 2018 

https://youtu.be/gm34Aph9v0M 

https://youtu.be/MAopsd6kdY0 

KubeCon Europe 2020: 

https://youtu.be/Yt5wpZ6raDA 

HashiTalks: Deploy 2021: 

https://youtu.be/PXVatM1xZxg

https://go.dok.community/slack

https://dok.community/ 

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)

The ability to extend Kubernetes with Custom Resource Definitions and respective controllers has led to the OperatorSDK, which became the de facto standard for data service automation on Kubernetes. There are countless operator implementations available, and new operators are being released on a daily basis. Organizations managing hundreds of Kubernetes clusters for dozens of developer teams are also challenged to manage the lifecycle of hundreds of Kubernetes operators. The goal is to keep the operational overhead to a minimum.

In this talk, a closer look into the lifecycle of operators will be presented. With an understanding of how operators evolve, it becomes clear what challenges during operator upgrades. A brief overview of lifecycle management tools such as Helm, OLM, and Carvel is presented in this context. In particular, it will be discussed whether these tools can help, which restrictions apply and where further development would be desirable. At the end of this talk, you will know what operator lifecycle management is about, what its challenges are, and which tools may be used to reduce operational friction.

Julian Fischer, CEO of anynines, has dedicated his career to the automation of software operations. In more than fifteen years, he has built several application platforms. He has been using Kubernetes, Cloud Foundry, and BOSH in recent years. Within platform automation, Julian has a strong focus on data service automation at scale.

https://go.dok.community/slack

https://dok.community/ 

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)

FoundationDB is an open-source distributed transactional Key-Value store that is used by multiple companies like Apple, Snowflake and VMWare Tanzu (previously Wavefront).

This talk will cover the design of the FoundationDB operator and lessons learned from operating FoundationDB on Kubernetes. We will discuss some of the missing pieces in Kubernetes to make it easier to operate FoundationDB on top of it and how we solved those challenges in the operator. We will focus on the pieces of the FoundationDB operator that are different to most other operators and why we decided to implement those pieces like they are. We will also discuss how to run an high available FoundationDB cluster on top of Kubernetes and what different choices a user has. We will also cover some challenges that arise when running stateful services at scale on top of Kubernetes and how they can be managed. At the end of this talk we will give an outlook for future design changes and planned features in our operator.

The main take-away from this talk is to understand how to run and operate FoundationDB on Kubernetes.

Johannes started his journey in the Kubernetes eco-system in early 2015, onboarding projects and applications onto Kubernetes. Since 2020 Johannes works as an SRE for FoundationDB at Apple and is co-leading the development of the open source FoundationDB operator.

https://go.dok.community/slack

https://dok.community/ 

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)

It is still challenging to run Apache Spark and other big data processing workload on Kubernetes, especially in large scale. People need to address various issues like resource isolation, queuing, and cost efficiency. This session will share details about those challenges and how to address them. We will also present a convenient (one-click) way to deploy Apache Spark on Kubernetes, and dramatically lower the barrier to use Spark.

Bo has been working in big data area for 10 years, and gained intensive experience of running Apache Spark in large scale while he worked in in Uber. In the Kubernetes era, he focuses on building world class Apache Spark Service, and making the tooling available to the Spark and Kubernetes communities.

https://go.dok.community/slack

https://dok.community/ 

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)

Data doesn’t magically appear in our data centers. There are usually several phases and several storage locations along its journey throughout your organization. New architectural patterns, such as microservices, and new technology, such as Kubernetes are changing how we can think about and manage the large volumes of data coming at us.

In this talk we will begin by quickly introducing the architecture and technology and how they make our lives better. From there it’s live demo time combining Java microservices, a processing caching service, a messaging layer, and a relational database all running in Kubernetes . This application will be handling frequently updated data, generating alerts on specific data events, and simultaneously populating a system of record. Come for the discussion, hands on demo, and witty banter!

Steve is a Dad, Partner, Son, and Senior Developer Readiness Engineer for VMware Tanzu. In addition to showing off all the great work of the Tanzu team, he helps drive Tanzu developer experience. He can teach you about Containers, Kubernetes, Data Analysis, Java, Python, PostgreSQL, Microservices, and some JavaScript. He has deep subject area expertise in GIS/Spatial, Statistics, and Ecology. Before Tanzu, Steve was a developer Advocate for Crunchy Data, DigitalGlobe, Red Hat, LinkedIn, deCarta, and ESRI. Steve has a Ph.D. in Ecology and can easily be bribed with offers of bird watching or fly fishing.

He has spoken at hundreds of conferences and done over 100 workshops including Monktoberfest, Red Hat Summit, MongoNY, JavaOne, FOSS4G, ODSC, AjaxWorld, GeoWeb, Where2.0, and OSCON.

https://go.dok.community/slack

https://dok.community/ 

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)

Kubernetes comes with a lot of useful features like Volumes and StatefulSets, which make running stateful workloads simple. Interestingly, when combined with the right tools, these features can make Kubernetes very valuable for developers wanting to run massive production databases in development! This is exactly what was seen at "Extendi".

The developers at Extendi deal with a large amount of data in their production Kubernetes clusters. But when developing locally, they didn't have an easy way of replicating this data. This replication was needed because it allowed developers to test new features instantaneously without worrying if they would work as expected when pushed to production. But replicating a 100Gb+ production database for development wasn't turning out to be an easy task!

This is where leveraging Kubernetes + remote development environments came to the rescue. Running data on Kubernetes turned out to be way faster than any of the traditional approaches because of Kubernetes' ability to handle stateful workloads exceptionally well. And since Extendi already used Kubernetes in production - the setup process was fairly simple.

This talk will cover practical steps on how leveraging Kubernetes based development environments allowed dev teams at Extendi to run production data on Kubernetes during development using features like Volume Snapshots, having a huge positive impact on developer productivity.

Arsh is a Developer Experience Engineer at Okteto. He is an active contributor to the upstream Kubernetes project and was awarded the Kubernetes Contributor Award for his contributions in 2021. Arsh has written blogs and spoken about different topics in the cloud-native ecosystem at various conferences before, including KubeCon + CloudNativeCon + Open Source Summit China 2021. He has also been on the Kubernetes Release Team since the 1.23 release. He also serves as the New Contributor Ambassador for the Documentation Special Interest Group of the Kubernetes project and continuously mentors new folks in the community. Previously, he worked at VMware and was an active contributor to other CNCF projects, including cert-manager and Kyverno.

Lapo is a Software Engineer currently leading the development team of a Social Listening and Audience Intelligence platform. He started coding at the early age of 14 and since he turned his passion into a real job, he has always been looking for boosting his knowledge by constantly researching for newer and newer technologies. Active on Ruby Open Source projects

Ramiro Berrelleza is one of the founders of Okteto. He has spent most of his career (and his free time) building cloud services and developer tools. Before starting Okteto, Ramiro was an Architect at Atlassian and a Software Engineer at Microsoft Azure. Originally from Mexico, he currently lives in San Francisco.

https://go.dok.community/slack

https://dok.community/ 

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)

Cloud-native applications comprise various components, including data services, storage systems, and related Kubernetes objects. Each component requires its own data protection tools, strategy, and domain expertise. A robust solution aligned with business requirements often involves complex workflows. What if there was a way to coordinate the implementation of these workflows while optimizing how backups are moved into storage?

During this talk, Pavan will demonstrate how two open-source tools, Kanister and Kopia, work together to optimize backup and recovery for Kubernetes applications.

This talk is for anyone interested in running stateful workloads in production on Kubernetes. Attendees will leave armed with a streamlined way to protect Kubernetes applications.

Pavan joined Kasten by Veeam in March 2018, where he manages a team of cloud-native engineers developing creative solutions for managing and protecting stateful application data in Kubernetes. Prior to Kasten, he was a part of NetApp’s Cloud Data Services Group, where he worked on the data protection stack for a cloud-integrated backup appliance. Pavan is an active contributor to open-source projects including Kanister, a framework for application-level data management on Kubernetes; and Kopia, a fast and secure backup tool. He holds a Master's degree in Computer Science from the University of Florida.

https://go.dok.community/slack

https://dok.community/ 

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)

It only looks simple to run databases in Kubernetes. In fact, it is too many things needed to be considered before running any database in Kubernetes. Failover and traffic switching, replication and data consistency/loss after failover, upgrades, DB and node-level configuration, CNI, backups, monitoring, etc. After this talk, you will have a complete list of questions that should be checked before running a database in production.

Mykola has been shorting “Time-to-market” in software companies for more than ten years. Most of his career he has been focused on the *development* of monitoring, update and deployment systems.

https://go.dok.community/slack

https://dok.community/ 

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)

How can you keep your data secure and how can you transfer them on a secure way? You will learn to encrypt your data, that you can use them Kubernetes based in a Multi Cloud environment.

Sarah Julia Kriesch is working as a DevOps Consultant for Accenture in Germany. She has studied Computer Science at the TH Nürnberg and worked as a Linux System Administrator/ System Engineer for an ISP and in the area of Cloud Computing before. She is also contributing to Open Source since around 10 years.

https://go.dok.community/slack

https://dok.community/ 

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)

Kubernetes is turning into a preferred platform for real-time analytic app that crunch billions of events per day and return insights in seconds. In this talk we'll introduce the standard analytic app design pattern of fast event streams coupled with low-latency data warehouses, using open source projects. We'll then walk through deploying the pipeline on Kubernetes from ingest to end user access. We'll touch on use of operators, scaling, monitoring, upgrade, security, and approaches to adding custom components. Attendees can expect to leave with concrete lessons about how to stand up low-latency analytics quickly on Kubernetes.

Robert Hodges is CEO of Altinity, an enterprise provider for ClickHouse data warehouse. He's also a database geek with experience on at least 20 DBMS types. Robert caught the Kubernetes bug at VMware in 2018.