Data on Kubernetes Community

https://dok.community/

Graph databases are fundamentally designed with a focus on the relationship between data sets. Running stateful workloads like databases on a container platform has always been a challenge. How to unleash the power of your connected data on Kubernetes with cloud-native technologies?

Nebula Graph is a highly performant linearly scalable graph database available for use via a shared-nothing distributed model. KubeSphere is an open source container platform built on Kubernetes, helping developers to implement application delivery and build observability with ease.

In this talk, maintainers from KubeSphere and Nebula Graph community will demonstrate how to leverage Operator to deploy and manage a graph database on K3s with KubeSphere.

- KubeSphere and K3s Walkthrough

- Graph Database and Nebula Graph: Learn and Practice

- How to use KubeKey to install K3s and OpenEBS within minutes

- Run Nebula Graph on K3s with KubeSphere

- Build cloud native nbservability for Database application

Feynman is a CNCF ambassador and community manager. He is growing and maintaining the KubeSphere open source community, which helps users to widely adopt Kubernetes and reduce the learning curve of using cloud-native technologies. He focuses on technical writing and advocacy and outreacy in cloud-native area.

https://dok.community/

ABSTRACT OF THE TALK

On-demand infrastructure services have become a key requirement for developers and DevOps engineers to provision and consume the resources they need. While it can be achieved in public cloud with the support of a rich API ecosystem, there's a lack of common framework across CSP's. As the next-generation Cloud Operating System, Kubernetes provides a variety of services that can be used to reach the same goal, but with a common framework across any type environment: on-premises, private or public clouds. In this talk, we're going to explore the solutions and patterns available for building end-to-end CI/CD pipelines that include persistent data services.

BIO

Nicolas is an experienced hands-on technologist, evangelist and product owner who has been working in the fields of Cloud-Native technologies, Open Source Software, Virtualization and Datacenter networking for the past 17 years.

Passionate about enabling users and building cool tech solving real-life problems, you'll often see him speaking at global tech conferences and online events, spreading the word and walking the walk with customers.

https://dok.community/

Spark and lakeFS are revolutionizing large scale data processing that is version-aware. Is it possible to run this architecture over Kubernetes? We’ll cover the fastest way to get this environment up and running, and the benefits you get with it. Finally we’ll show how horizontal scaling and the lakeFS Hadoop Filesystem avoid processing bottlenecks as workloads increase.

Itai is a R&D team leader at Treeverse, the company behind open-source lakeFS. He thrives on finding creative solutions for complex problems, especially if it involves code. Previously, Itai worked at Microsoft and Ridge on data infrastructure, tooling, and performance. Itai received his B.Sc degree in Computer Science and an MBA from Tel Aviv University.

- Importance of building reproducible data pipelines.

- Managing your data the same way you're managing your code.

https://dok.community/

A few short years ago I was a huge opponent of putting any data in Kubernetes, now I'm one of the most vocal proponents of doing so. Listen to how the ecosystem changed my mind.

An experienced technologist who has been hands-on in solving data problems for as long as I can remember. Currently working on changing the way we use storage in the datacenter.

https://dok.community/

CAPE is an open source solution offering advanced kubernetes features for DR, data migration and mobility, multi-cluster app deployment etc. I will demo how to install CAPE and also cover how to backup and restore data using CAPE

Learn about a new open source tool for kubernetes backup/restore

I come from a strong infrastructure background with expertise in infrastructure automation, containers, kubernetes, DevOps and service mesh.

https://dok.community/

How to build a stateful, transactional streaming storage engine on top of kubernetes.

1. Building storage engines in kubernetes is hard

2. You need tiered storage (S3 or google cloud bucket, Azure Blob) for total disaster recovery

3. You also need sound replication inside the cluster

4. You need a simple architecture so you can scale the pods easily

Alex Gallego is the founder and CEO of Vectorized, where he & the team hack on Redpanda, a modern streaming platform for mission critical workloads. Prior to Vectorized, he was a principal engineer at Akamai, as well as co-founder and CTO of Concord.io, a high performance stream processing engine built in C++ and acquired by Akamai in 2016.

https://dok.community/

ETL/ELT on Kubernetes is currently an unsolved problem. There are a lot of different approaches vying for a spot as the de facto method, but none are clear winners. Considering that the cloud-native landscape is built for deploying Dockerized, open-source software, many of the closed-source solutions fall flat and don't mesh with the trajectory of the community.

Airbyte is an open-source ETL/ELT tool that harmonizes well with the cloud-native landscape and lives to enable your stateful workloads on Kubernetes. Previously, I have talked about a theoretical deployment on Kubernetes and the nuances behind deploying an ETL/ELT pipeline in such an environment. Now, I'm looking to follow that up with how we actually implemented that strategy as we launched our K8s beta. Additionally, I'll dive into some of the nitty gritty details that we needed to figure out in order to get this all working... stuff that isn't really found online!

Overall, this will be a really unique case of getting to do a retrospective on what we planned our architecture to look like and following up with some great development insights as we solidified the final implementation.

- Quick overview of Airbyte and open-source ETL/ELT [5 minutes]

- Why run your ETL/ELT in K8s? [3 minutes]

- A quick recap on the previous talk (what we thought the architecture would look like) [5 minutes]

- Display the actual architecture and implementation [10 minutes]

-> Talk about how to communicate with k8s pods on STDOUT and STDIN pipes

-> Describe parent-child process termination strategy

-> Describe persistence layer/strategy and config storage

- Quick demo of an Airbyte deployment on K8s [10 minutes]

Abhi is a confused economist who enjoys writing backend code for data management software. He now spends most of his time doing developer relations in the data integration space, where he looks to evangelize open source technologies. In his spare time he is a DJ, drummer, and competitive Super Smash Bros. player. He is a staunch advocate of proper semicolon usage, Oxford commas and overused grammar jokes.

https://dok.community/

With the help of 3rd parties, running stateful workloads can be a “trivial” experience but what happens when you need to migrate between vendors?

Civo offers managed Kubernetes clusters powered by K3s to customers around the world. We manage thousands of Virtual Machines and stateful customer data within multiple data centres across the globe. During recent vendor contract renewal negotiations, it was decided to move to a new storage provider. This required the team to find a way of having the 2 storage vendors co-exist on existing superclusters, and have the ability to safely migrate data between the 2 vendors.

Join Civo’s Director of Innovation, Dinesh, as he talks through the research, testing and migration process carried out in the past few months.

Dinesh is Director of Innovation at Civo and has been the lead architect for the new Civo Stack Platform. Having worked in the hosting industry for many years, Dinesh has a passion for creating solutions that operate at scale. This not only applies to the technology stack, but for nurturing engineers through their career.

https://dok.community/

Cassandra is a highly scalable database with an architecture that makes it well suited for multi-region workloads. A Kubernetes cluster often spans across multiple zones within a single region. Multi-region Kubernetes clusters are less common though due to the challenges that they present. This has led to a growing number of multi-cluster solutions. In this presentation John Sanda introduces K8ssandra Operator. It is designed from the ground up for multi-cluster deployments. John will discuss how to reconcile objects across multiple clusters, how to manage secrets, pitfalls to avoid, and testing strategies.

John Sanda is a DataStax engineer working on the K8ssandra project. He is passionate about Cassandra and Kubernetes and loves being involved in open source. Prior to joining DataStax John worked for a year at The Last Pickle as an Apache Cassandra consultant. Prior to that, he spent a number of years at Red Hat as an engineer. It was during that time John got involved with Cassandra when he redesigned a metrics data store and built it with Cassandra in place of an RDBMS. He had his first initial exposure to Cassandra and Kubernetes when the metrics storage engine was later used in OpenShift.

https://dok.community/

There is confusion amongst Kubernetes users as to when you should make a Deployment with a PVC and when you should use a StatefulSet with a PVC. There is also a general lack of understanding when it comes to disk access policies, what RWO/RWX mean and what they allow you to do. These concepts are complicated and require a deep level of understanding in order to avoid users making bad decisions that they come to regret later.

In this talk we will explore when you should use each type and what things you need to think about before making a decision. We'll also explore in detail how to safely deploy a multi-replica application with persistence using Portainer.

As part of the talk we will explore disk access policy, understand what RWO really is and how RWX disk access changes the equation in regards to persistence.

From this talk delegates will learn the difference between persistent disk and StatefulSet. They will learn when to use each (and when not to) and what are the pitfalls they should be aware of when deploying multi replica apps. They will also learn what is disk access policy, what is RWO and how RWX disk access changes the equation in regards to persistence

Neil is the CEO and Co-Founder of Portainer.io, based in Auckland New Zealand.

Neil is a career consultant, having spent 25 years in the roles of systems engineering, IT consulting, and more recently IT management. Neil started his career with 12 years at IBM as his foundation, where he had a leading role in Server, Storage, and Virtualisation systems engineering.

Neil moved from IBM into self-employment with two business partners, who together created one of NZ’s (and later ASEANs) leading VMware consulting practices, ViFX. During this time, Neil was based in Singapore, and was responsible for the design and deployment of the largest and most complex VMware deployments in Asia Pacific. This experience saw Neil consulting to CIOs, CISO’s and CTO’s all over the region, picking up a vast array of intelligence on what these senior IT leaders expect from a product before it can obtain widespread adoption in their organisations.

Neil then moved into a CEO role at a start-up Cloud Service Provider, whereby he grew that business from nothing to $8M ARR in just 2 years.

Neil’s exposure to Docker technologies came 4 years ago, when at the helm of the Cloud Service Provider, he wanted to bring a “Containers as a Service” offering to market (well before many of the hyper-scales had such offerings). Neil attempted to find self-service portals that could be deployed by the service provider as a way to help his customers consume the service, but there were none available, so Neil had Portainer built.

Portainer has now grown into an entity of its own right, and using Neil’s knowledge of the CIO’s requirements, knowledge of the technology, and knowledge of infrastructure management, Neil has crafted Portainer into a strong contender in the Container management space.

Neil now runs the Portainer.io business, and is responsible for the overall product strategy and the team entrusted to deliver on that strategy.