Data on Kubernetes Community

https://go.dok.community/slack

https://dok.community/ 

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)

Multiple clusters exist in most Kubernetes environments today, and number of clusters will increase overtime. The reasons for having multiple Kubernetes clusters are many, for example, overcoming scale limits, reducing complexity, geo separation, redundancy and having separate production, staging, and development environments. Once you have multiple K8S clusters, it can be useful to have the ability to easily move or duplicate workloads across these different clusters. Kubernetes does not have a native method to allow migration or duplication of workloads across clusters.

Fortunately, there are tools that provide this functionality. In this presentation we will explore the different uses cases for cross cluster migration, and what is involved, and how these migration tools work. We'll cover some popular uses cases, such as, Disaster Recovery, Test/Dev, and performance testing. Migration could entail moving the entire cluster, or individual workloads. The components that need to be moved would include configuration and resources stored in etcd, and persistent data residing on PVCs. We'll cover the uses cases and challenges for migration, and run through an example of using one of these migration tools.

https://go.dok.community/slack

https://dok.community/ 

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)

Some data-intensive workloads are easier to run in Kubernetes than others. Why? What needs to improve? Join us as we deep dive with Adobe and the CNCF about how easy (or not) it is to run different types of data workloads on Kubernetes – and what is being done both inside and outside of Kubernetes to make data workloads easier.

Xing Yang is a Tech Lead in the Cloud Native Storage team at VMware. She is a co-chair of the CNCF TAG Storage, a co-chair of the Kubernetes SIG Storage, a co-chair of the Kubernetes Data Protection WG, and a maintainer in Kubernetes CSI. Before joining VMware, Xing was the Lead Architect of OpenSDS while working for Futurewei. She also worked at Dell EMC for many years and has developed deep expertise in storage, data protection, disaster recovery, cloud, and virtualization technologies.

I find myself working at the intersection of OpenSource, Cloud Native software, SRE, and DevOps. My goal is to help bring a pragmatic opinion and shepherd organizations to stay focused on delivering value to their internal and external customers

https://go.dok.community/slack

https://dok.community/ 

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)

We know from the first Data on Kubernetes Report that 90% of respondents believe Kubernetes is ready for stateful workloads, but significant challenges remain. The DoK Community continues to grow and build a unique space where people share knowledge and have conversations that are shaping the next decade of data on Kubernetes.

Melissa Logan and Sylvain Kalache will discuss the growth of DoK, DoKC, and introduce a new project to help end users on their journey to running data-intensive workloads on Kubernetes more easily and with better control.

https://go.dok.community/slack

https://dok.community/ 

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)

The state of the CNCF Storage options has exploded in the past few years, but if you had to choose a project to use today, how would you go about comparing each offering and choosing who to partner with for your future growth?

Civo offers managed Kubernetes clusters powered by K3s to customers around the world. We manage thousands of Virtual Machines and stateful customer data within multiple data centres across several continents. In late 2021, Civo had the opportunity to evaluate the CNCF storage landscape to move to a new technology stack.

Learn about the steps required to evaluate the market, testing vendor claims in the real world and what needs to be considered outside of the purely technological aspects of any product selection.

In late 2021, Civo's Director of Innovation, Dinesh, spoke about the intricacies of migrating between two vendors (https://www.youtube.com/watch?v=GlUsPnSZI_Y&list=PLHgdNuGxrJt2-xlW_l2q1BAE3e4TsPnQo). Now join him as he talks about the wider project and some of business implications of hosting Data on Kubernetes

Dinesh is Director of Innovation at Civo and has been the lead architect for the new Civo Stack Platform. Having worked in the hosting industry for many years, Dinesh has a passion for creating solutions that operate at scale. This not only applies to the technology stack, but for nurturing engineers through their career.

https://go.dok.community/slack

https://dok.community/ 

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)

The Rap God project acts as a great entry point to many incoming open-source enthusiasts who are interested in learning about the cloud native ecosystem. The Rap-God project uses Kubernetes orchestration for a stateful case which is an emerging topic, the Rap God project acts as a demonstration of how to use such features of Kubernetes. The project will be using Stateful sets that’d deploy Apache Cassandra (for its first cycle) and eventually it’ll be implementing the same API endpoints for various databases that will be with/on Kubernetes. We in the community intend to do this with PersistenceVolumes and Persistent Volume Claims. Keeping in mind the issues, various developers face, we also will be making options for storage classes. The project will allow the members to explore how they can customize the whole storage class setup according to their setup. The project will be bringing Helm, Cassandra, Kubernetes and Argo under its watch and shall actively expand on its implementation with the further iterations.

Abhijith Ganesh is an undergrad computer science major, currently pursuing his Freshman year. His areas of interest include DevOps, Kuberenetes and Open Source Projects.

He is an active member of the DoK Community where he is currently an intern. He is also member of the Pyrsia and SeaQL communities.

https://go.dok.community/slack

https://dok.community/ 

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)

We will walk through how to build serverless event streaming applications as functions running in a function mesh on kubernetes with cloud native messaging via Apache Pulsar.

In this talk, you will deploy ML functions to transform real-time data on Kubernets.

Tim Spann is a Developer Advocate @ StreamNative where he works with Apache Pulsar, Apache Flink, Apache NiFi, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Principal Field Engineer at Cloudera, a Senior Solutions Architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science.

https://www.datainmotion.dev/p/about-me.html 

https://dzone.com/users/297029/bunkertor.html 

https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/speaker/185963

https://go.dok.community/slack

https://dok.community/ 

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)

Adobe runs dozens of Kafka clusters spread across both public (AWS and Azure) and private clouds to power the Adobe Experience Platform message bus.

In this talk, we share the experience of running Kafka as a stateful Kubernetes workload in a cloud-native environment using the open-source Koperator and the tools we built to automate the whole lifecycle.

While running a stateful service like Kafka on Kubernetes may be intimidating at the first glance, we share our thought process, the tools, and the results that can make this a reality in any organization. The Kubernetes Operator pattern helped us automate all the operational aspects for the lifecycle of the cluster; abstract away the cloud specifics allowing us to focus on Kafka; achieve increased resilience and elasticity; implement automated Kafka rebalancing using CruiseControl, and harness all the metrics to implement an observable environment. We also plan to demo how these all come together.

Join us as we share our team's learnings on:

If you or your team ever wanted to learn how to run your own stateful service in Kubernetes and/or see what's possible, come join us!

Computer Science Ph.D. and Principal Scientist at Adobe having experience in complex, large scale distributed systems.

My focus currently is on designing and building the message streaming platform for Adobe Experience Platform using Apache Kafka. The endeavor is unique as it spans both Amazon AWS and Microsoft Azure clouds and handles billions of messages a day.

https://go.dok.community/slack

https://dok.community/ 

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)

Running a database on Kubernetes with persistent storage is relatively easy but when it comes to performance it won’t match local NVMes. This talk will show you how to set up the local NVMes for Kubernetes, how to handle the application and cluster lifecycle in a safe manner and share our experience with running ScyllaDB with local NVMes on different Kubernetes cloud providers.

Tomas leads the development of Scylla Operator (https://github.com/scylladb/scylla-operator), a Kubernetes operator to manage ScyllaDB. Previously, he worked on a self-hosted, auto-upgrading Kubernetes control plane for RedHat OpenShift. Tomas is an Emeritus Kubernetes SIG-Apps approver.

Maciej is a Go and C++ enthusiast. He is a software engineer working on ScyllaDB management tools. Previously he worked in network companies where he delivered multiple features to SDN solutions and LTE networks.

https://go.dok.community/slack

https://dok.community/ 

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)

Redis is a widely used open-source in-memory data store and cache that has become a key component in the development of scalable microservice systems. While all of the main cloud providers provide fully managed Redis services (Amazon ElastiCache, Azure Cache for Redis, and GCP Memorystore), it may also be simply implemented in Kubernetes if you require additional control over the Redis configurations.

This session will lead you through setting up a Redis cluster on Kubernetes with data persistence. The lesson would concentrate on establishing a "prod"-ready Redis cluster that considers storage and persistence even after restarts and discusses recommended practices and considerations.

With the help of monitoring and hardware optimizations for your cluster, you'll learn about measuring your performance and best practices for Redis on Kubernetes. We would conclude whether operators or Helm chart will be a more mature option for your Data on Kubernetes needs with Redis and a few tips as a production checklist.

Ryan Gray is a Technical Consultant with Portainer.io. Before that he spent a couple of decades building ISPs, carrier Ethernet networks, data centers, and software. He is currently obsessed with container orchestration, GitOps, edge computing, and why naming things is so hard.

Hrittik is a Community Advocate at Portainer and a student in his sophomore year. Hrittik focuses on building tech communities and software with the help of cloud native technologies. He has been a speaker at conferences like the Cloud Summit and conducted various events that have collectively empowered students and professionals on DevOps and Cloud.

https://go.dok.community/slack

https://dok.community/ 

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)

Accidental PVC delete or namespace delete can cause the Persistent Volume to get deleted. Such volumes lose their data and the stateful applications lose their state. By the use of Persistent Volume TrashCan, users can get a grace period to undo such unintended delete operation.

The deleted Persistent Volumes are staged for delayed deletes. They continue to live even after being deleted from k8 perspective, for a configurable time(retention period) and based on the system’s usage. The storage class of the PVC can dictate if they need to be staged for a delayed delete. StorageClass can also allow for configurable retention period.

To recover a deleted PersistentVolume, users can create a new namespace with the same name and reapply the original PVC spec. The PVC will reference a special StorageClass to indicate that the new PersistentVolume needs to be restored from the TrashCan. This will allow the application to restart with the right state and data.

This talk will showcase how to overcome one of the admin’s pain point seen in field involving accidental deletions of PVCs by using advanced storage management solutions in Kubernetes.

Veda Talakad is a Software professional with BS in electronics and communications mostly working in storage domain. Some of the professional areas of interests include scalability in distributed environment and cloud-native technologies for data management,

Aditya Kulkarni: I have 10 years of experiance in Enterprise Data management. I have worked on All Flash Array at Netapp from the device driver layer to WAFL Filesystem. Next, I worked at Portworx where I developed and enhanced the number one data platform for Kubernetes. Then I moved to Pure Storage as part of Portworx acquisition.

Aditya Dani is an architect at Portworx by PureStorage, that enables users to run any cloud-native data service, in any cloud, using any Kubernetes platform. He is one of the contributors to the open source project Stork that provides storage awareness to the Kubernetes scheduler. Prior to Portworx, Aditya worked for Amazon on their Music Recommendations Engine. He enjoys working on distributed systems and loves programming in Go.