Data Engineering Podcast

StreamNative Brings Streaming Data To The Cloud Native Landscape With Pulsar


Listen Later

Summary

There have been several generations of platforms for managing streaming data, each with their own strengths and weaknesses, and different areas of focus. Pulsar is one of the recent entrants which has quickly gained adoption and an impressive set of capabilities. In this episode Sijie Guo discusses his motivations for spending so much of his time and energy on contributing to the project and growing the community. His most recent endeavor at StreamNative is focused on combining the capabilities of Pulsar with the cloud native movement to make it easier to build and scale real time messaging systems with built in event processing capabilities. This was a great conversation about the strengths of the Pulsar project, how it has evolved in recent years, and some of the innovative ways that it is being used. Pulsar is a well engineered and robust platform for building the core of any system that relies on durable access to easily scalable streams of data.

Announcements
  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, a 40Gbit public network, fast object storage, and a brand new managed Kubernetes platform, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. And for your machine learning workloads, they’ve got dedicated CPU and GPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
  • You monitor your website to make sure that you’re the first to know when something goes wrong, but what about your data? Tidy Data is the DataOps monitoring platform that you’ve been missing. With real time alerts for problems in your databases, ETL pipelines, or data warehouse, and integrations with Slack, Pagerduty, and custom webhooks you can fix the errors before they become a problem. Go to dataengineeringpodcast.com/tidydata today and get started for free with no credit card required.
  • Your host is Tobias Macey and today I’m interviewing Sijie Guo about the current state of the Pulsar framework for stream processing and his experiences building a managed offering for it at StreamNative
  • Interview
    • Introduction
    • How did you get involved in the area of data management?
    • Can you start by giving an overview of what Pulsar is?
      • How did you get involved with the project?
      • What is Pulsar’s role in the lifecycle of data and where does it fit in the overall ecosystem of data tools?
      • How has the Pulsar project evolved or changed over the past 2 years?
        • How has the overall state of the ecosystem influenced the direction that Pulsar has taken?
        • One of the critical elements in the success of a piece of technology is the ecosystem that grows around it. How has the community responded to Pulsar, and what are some of the barriers to adoption?
          • How are you and other project leaders addressing those barriers?
          • You were a co-founder at Streamlio, which was built on top of Pulsar, and now you have founded StreamNative to offer Pulsar as a service. What did you learned from your time at Streamlio that has been most helpful in your current endeavor?
            • How would you characterize your relationship with the project and community in each role?
            • What motivates you to dedicate so much of your time and enery to Pulsar in particular, and the streaming data ecosystem in general?
              • Why is streaming data such an important capability?
              • How have projects such as Kafka and Pulsar impacted the broader software and data landscape?
              • What are some of the most interesting, innovative, or unexpected ways that you have seen Pulsar used?
              • When is Pulsar the wrong choice?
              • What do you have planned for the future of StreamNative?
              • Contact Info
                • LinkedIn
                • @sijieg on Twitter
                • sijie on GitHub
                • Parting Question
                  • From your perspective, what is the biggest gap in the tooling or technology for data management today?
                  • Closing Announcements
                    • Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
                    • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
                    • If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
                    • To help other people find the show please leave a review on iTunes and tell your friends and co-workers
                    • Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
                    • Links
                      • Apache Pulsar
                        • Podcast Episode
                        • StreamNative
                        • Streamlio
                        • Hadoop
                        • HBase
                        • Hive
                        • Tencent
                        • Yahoo
                        • BookKeeper
                        • Publish/Subscribe
                        • Kafka
                        • Zookeeper
                          • Podcast Episode
                          • Kafka Connect
                          • Pulsar Functions
                          • Pulsar IO
                          • Kafka On Pulsar
                            • Webinar Video
                            • Pulsar Protocol Handler
                            • OVH Cloud
                            • Open Messaging
                            • ActiveMQ
                            • Kubernetes
                            • Helm
                            • Pulsar Helm Charts
                            • Grafana
                            • BestPay(?)
                            • Lambda Architecture
                            • Event Sourcing
                            • WebAssembly
                            • Apache Flink
                              • Podcast Episode
                              • Pulsar Summit
                              • The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

                                Support Data Engineering Podcast

                                ...more
                                View all episodesView all episodes
                                Download on the App Store

                                Data Engineering PodcastBy Tobias Macey

                                • 4.5
                                • 4.5
                                • 4.5
                                • 4.5
                                • 4.5

                                4.5

                                142 ratings


                                More shows like Data Engineering Podcast

                                View all
                                Software Engineering Radio by se-radio@computer.org

                                Software Engineering Radio

                                271 Listeners

                                The Changelog: Software Development, Open Source by Changelog Media

                                The Changelog: Software Development, Open Source

                                289 Listeners

                                Data Skeptic by Kyle Polich

                                Data Skeptic

                                479 Listeners

                                Software Engineering Daily by Software Engineering Daily

                                Software Engineering Daily

                                625 Listeners

                                Talk Python To Me by Michael Kennedy

                                Talk Python To Me

                                585 Listeners

                                Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

                                Super Data Science: ML & AI Podcast with Jon Krohn

                                302 Listeners

                                CoRecursive: Coding Stories by Adam Gordon Bell - Software Developer

                                CoRecursive: Coding Stories

                                190 Listeners

                                DataFramed by DataCamp

                                DataFramed

                                269 Listeners

                                Practical AI by Practical AI LLC

                                Practical AI

                                211 Listeners

                                AWS Podcast by Amazon Web Services

                                AWS Podcast

                                203 Listeners

                                The Stack Overflow Podcast by The Stack Overflow Podcast

                                The Stack Overflow Podcast

                                63 Listeners

                                The Real Python Podcast by Real Python

                                The Real Python Podcast

                                142 Listeners

                                Last Week in AI by Skynet Today

                                Last Week in AI

                                306 Listeners

                                This Day in AI Podcast by Michael Sharkey, Chris Sharkey

                                This Day in AI Podcast

                                227 Listeners

                                The Pragmatic Engineer by Gergely Orosz

                                The Pragmatic Engineer

                                64 Listeners