Data Engineering Podcast

Wallaroo with Sean T. Allen - Episode 12

Listen Later
Summary

Data oriented applications that need to operate on large, fast-moving sterams of information can be difficult to build and scale due to the need to manage their state. In this episode Sean T. Allen, VP of engineering for Wallaroo Labs, explains how Wallaroo was designed and built to reduce the cognitive overhead of building this style of project. He explains the motivation for building Wallaroo, how it is implemented, and how you can start using it today.

Preamble
  • Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure
  • When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at dataengineeringpodcast.com/linode and get a $20 credit to try out their fast and reliable Linux virtual servers for running your data pipelines or trying out the tools you hear about on the show.
  • Continuous delivery lets you get new features in front of your users as fast as possible without introducing bugs or breaking production and GoCD is the open source platform made by the people at Thoughtworks who wrote the book about it. Go to dataengineeringpodcast.com/gocd to download and launch it today. Enterprise add-ons and professional support are available for added peace of mind.
  • Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.
  • You can help support the show by checking out the Patreon page which is linked from the site.
  • To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers
  • Your host is Tobias Macey and today I’m interviewing Sean T. Allen about Wallaroo, a framework for building and operating stateful data applications at scale
    • Interview
    • Introduction
    • How did you get involved in the area of data engineering?
    • What is Wallaroo and how did the project get started?
    • What is the Pony language, and what features does it have that make it well suited for the problem area that you are focusing on?
    • Why did you choose to focus first on Python as the language for interacting with Wallaroo and how is that integration implemented?
    • How is Wallaroo architected internally to allow for distributed state management?
      • Is the state persistent, or is it only maintained long enough to complete the desired computation?
      • If so, what format do you use for long term storage of the data?


      • What have been the most challenging aspects of building the Wallaroo platform?

      • Which axes of the CAP theorem have you optimized for?

      • For someone who wants to build an application on top of Wallaroo, what is involved in getting started?

      • Once you have a working application, what resources are necessary for deploying to production and what are the scaling factors?

        • What are the failure modes that users of Wallaroo need to account for in their application or infrastructure?


        • What are some situations or problem types for which Wallaroo would be the wrong choice?

        • What are some of the most interesting or unexpected uses of Wallaroo that you have seen?

        • What do you have planned for the future of Wallaroo?

          • Contact Info
          • IRC
          • Mailing List
          • Wallaroo Labs Twitter
          • Email
          • Personal Twitter
            • Parting Question
            • From your perspective, what is the biggest gap in the tooling or technology for data management today?
              • Links
              • Wallaroo Labs
              • Storm Applied
              • Apache Storm
              • Risk Analysis
              • Pony Language
              • Erlang
              • Akka
              • Tail Latency
              • High Performance Computing
              • Python
              • Apache Software Foundation
              • Beyond Distributed Transactions: An Apostate’s View
              • Consistent Hashing
              • Jepsen
              • Lineage Driven Fault Injection
              • Chaos Engineering
              • QCon 2016 Talk
              • Codemesh in London: How did I get here?
              • CAP Theorem
              • CRDT
              • Sync Free Project
              • Basho
              • Wallaroo on GitHub
              • Docker
              • Puppet
              • Chef
              • Ansible
              • SaltStack
              • Kafka
              • TCP
              • Dask
              • Data Engineering Episode About Dask
              • Beowulf Cluster
              • Redis
              • Flink
              • Haskell

                • The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

                Support Data Engineering Podcast

                ...more
                View all episodesView all episodes
                Download on the App Store
                Data Engineering PodcastBy Tobias Macey
                • 4.6
                • 4.6
                • 4.6
                • 4.6
                • 4.6

                4.6

                135 ratings

                More shows like Data Engineering Podcast

                View all
                Software Engineering Radio - the podcast for professional software developers by se-radio@computer.org

                Software Engineering Radio - the podcast for professional software developers

                272 Listeners

                The Changelog: Software Development, Open Source by Changelog Media

                The Changelog: Software Development, Open Source

                283 Listeners

                The Cloudcast by Massive Studios

                The Cloudcast

                153 Listeners

                Thoughtworks Technology Podcast by Thoughtworks

                Thoughtworks Technology Podcast

                41 Listeners

                Data Skeptic by Kyle Polich

                Data Skeptic

                483 Listeners

                Talk Python To Me by Michael Kennedy

                Talk Python To Me

                592 Listeners

                Software Engineering Daily by Software Engineering Daily

                Software Engineering Daily

                624 Listeners

                The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) by Sam Charrington

                The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

                444 Listeners

                Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

                Super Data Science: ML & AI Podcast with Jon Krohn

                298 Listeners

                Python Bytes by Michael Kennedy and Brian Okken

                Python Bytes

                213 Listeners

                DataFramed by DataCamp

                DataFramed

                266 Listeners

                Practical AI by Practical AI LLC

                Practical AI

                190 Listeners

                The Stack Overflow Podcast by The Stack Overflow Podcast

                The Stack Overflow Podcast

                64 Listeners

                The Real Python Podcast by Real Python

                The Real Python Podcast

                140 Listeners

                Latent Space: The AI Engineer Podcast by swyx + Alessio

                Latent Space: The AI Engineer Podcast

                77 Listeners