Data Engineering Podcast

Understanding The Immune System With Data At ImmunAI


Listen Later

Summary

The life sciences as an industry has seen incredible growth in scale and sophistication, along with the advances in data technology that make it possible to analyze massive amounts of genomic information. In this episode Guy Yachdav, director of software engineering for ImmunAI, shares the complexities that are inherent to managing data workflows for bioinformatics. He also explains how he has architected the systems that ingest, process, and distribute the data that he is responsible for and the requirements that are introduced when collaborating with researchers, domain experts, and machine learning developers.

Announcements
  • Hello and welcome to the Data Engineering Podcast, the show about modern data management
  • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
  • RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. With RudderStack you can use all of your customer data to answer more difficult questions and then send those insights to your whole customer data stack. Sign up free at dataengineeringpodcast.com/rudder today.
  • Today’s episode is Sponsored by Prophecy.io – the low-code data engineering platform for the cloud. Prophecy provides an easy-to-use visual interface to design & deploy data pipelines on Apache Spark & Apache Airflow. Now all the data users can use software engineering best practices – git, tests and continuous deployment with a simple to use visual designer. How does it work? – You visually design the pipelines, and Prophecy generates clean Spark code with tests on git; then you visually schedule these pipelines on Airflow. You can observe your pipelines with built in metadata search and column level lineage. Finally, if you have existing workflows in AbInitio, Informatica or other ETL formats that you want to move to the cloud, you can import them automatically into Prophecy making them run productively on Spark. Create your free account today at dataengineeringpodcast.com/prophecy.
  • Your host is Tobias Macey and today I’m interviewing Guy Yachdav, Director of Software Engineering at Immunai, about his work at Immunai to wrangle biological data for advancing research into the human immune system.
  • Interview
    • Introduction (see Guy’s bio below)
    • How did you get involved in the area of data management?
    • Can you describe what Immunai is and the story behind it?
    • What are some of the categories of information that you are working with?
      • What kinds of insights are you trying to power/questions that you are trying to answer with that data?
      • Who are the stakeholders that you are working with and how does that influence your approach to the integration/transformation/presentation of the data?
      • What are some of the challenges unique to the biological data domain that you have had to address?
        • What are some of the limitations in the off-the-shelf tools when applied to biological data?
        • How have you approached the selection of tools/techniques/technologies to make your work maintainable for your engineers and accessible for your end users?
        • Can you describe the platform architecture that you are using to support your stakeholders?
          • What are some of the constraints or requirements (e.g. regulatory, security, etc.) that you need to account for in the design?
          • What are some of the ways that you make your data accessible to AI/ML engineers?
          • What are the most interesting, innovative, or unexpected ways that you have seen Immunai used?
          • What are the most interesting, unexpected, or challenging lessons that you have learned while working at Immunai?
          • What do you have planned for the future of the Immunai data platform?
          • Contact Info
            • LinkedIn
            • Parting Question
              • From your perspective, what is the biggest gap in the tooling or technology for data management today?
              • Closing Announcements
                • Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
                • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
                • If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
                • To help other people find the show please leave a review on iTunes and tell your friends and co-workers
                • Links
                  • ImmunAI
                  • Apache Arrow
                  • Columbia Genome Center
                  • Dagster
                    • Podcast Episode
                    • The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

                      Support Data Engineering Podcast

                      ...more
                      View all episodesView all episodes
                      Download on the App Store

                      Data Engineering PodcastBy Tobias Macey

                      • 4.6
                      • 4.6
                      • 4.6
                      • 4.6
                      • 4.6

                      4.6

                      135 ratings


                      More shows like Data Engineering Podcast

                      View all
                      Software Engineering Radio - the podcast for professional software developers by se-radio@computer.org

                      Software Engineering Radio - the podcast for professional software developers

                      272 Listeners

                      The Changelog: Software Development, Open Source by Changelog Media

                      The Changelog: Software Development, Open Source

                      282 Listeners

                      The Cloudcast by Massive Studios

                      The Cloudcast

                      152 Listeners

                      Thoughtworks Technology Podcast by Thoughtworks

                      Thoughtworks Technology Podcast

                      42 Listeners

                      Data Skeptic by Kyle Polich

                      Data Skeptic

                      481 Listeners

                      Talk Python To Me by Michael Kennedy

                      Talk Python To Me

                      591 Listeners

                      Software Engineering Daily by Software Engineering Daily

                      Software Engineering Daily

                      627 Listeners

                      The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) by Sam Charrington

                      The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

                      440 Listeners

                      Super Data Science: ML & AI Podcast with Jon Krohn by Jon Krohn

                      Super Data Science: ML & AI Podcast with Jon Krohn

                      298 Listeners

                      Python Bytes by Michael Kennedy and Brian Okken

                      Python Bytes

                      213 Listeners

                      DataFramed by DataCamp

                      DataFramed

                      265 Listeners

                      Practical AI by Practical AI LLC

                      Practical AI

                      189 Listeners

                      The Stack Overflow Podcast by The Stack Overflow Podcast

                      The Stack Overflow Podcast

                      64 Listeners

                      The Real Python Podcast by Real Python

                      The Real Python Podcast

                      140 Listeners

                      Latent Space: The AI Engineer Podcast by swyx + Alessio

                      Latent Space: The AI Engineer Podcast

                      76 Listeners