Data Engineering Podcast

By Tobias MaceyTechnology Education

Favorites
137
Followers
Typical duration
56 min
per episode

Based on Podcast App listening data

Data Engineering Podcast episodes

November 10, 2021Eliminate Friction In Your Data Platform Through Unified Metadata Using OpenMetadata
Summary
A significant source of friction and wasted effort in building and integrating data management systems is the fragmentation of metadata across various tools. After experiencing the impacts of fragmented metadata and previous attempts at building a solution Suresh Srinivas and Sriharsha Chintalapani created the OpenMetadata project. In this episode they share the lessons that they have learned through their previous attempts and the positive impact that a unified metadata layer had during their time at Uber. They also explain how the OpenMetadat project is aiming to be a common standard for defining and storing metadata for every use case in data platforms and the ways that they are architecting the reference implementation to simplify its adoption. This is an ambitious and exciting project, so listen and try it out today.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Struggling with broken pipelines? Stale dashboards? Missing data? If this resonates with you, you’re not alone. Data engineers struggling with unreliable data need look no further than Monte Carlo, the world’s first end-to-end, fully automated Data Observability Platform! In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Monte Carlo monitors and alerts for data issues across your data warehouses, data lakes, ETL, and business intelligence, reducing time to detection and resolution from weeks or days to just minutes. Start trusting your data with Monte Carlo today! Visit dataengineeringpodcast.com/impact today to save your spot at IMPACT: The Data Observability Summit a half-day virtual event featuring the first U.S. Chief Data Scientist, founder of the Data Mesh, Creator of Apache Airflow, and more data pioneers spearheading some of the biggest movements in data. The first 50 to RSVP with this link will be entered to win an Oculus Quest 2 — Advanced All-In-One Virtual Reality Headset. RSVP today – you don’t want to miss it!
Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads? Hightouch is the easiest way to sync data into the platforms that your business teams rely on. The data you’re looking for is already in your data warehouse and BI tools. Connect your warehouse to Hightouch, paste a SQL query, and use their visual mapper to specify how data should appear in your SaaS systems. No more scripts, just SQL. Supercharge your business teams with customer data using Hightouch for Reverse ETL today. Get started for free at dataengineeringpodcast.com/hightouch.
Your host is Tobias Macey and today I’m interviewing Sriharsha Chintalapani and Suresh Srinivas about OpenMetadata, an open standard for metadata and a reference implementation for a central metadata store
Interview
Introduction
How did you get involved in the area of data management?
Can you describe what the OpenMetadata project is and the story behind it?
What are the goals of the project?
What are the common challenges faced by engineers and data practitioners in organizing the metadata for their systems?
What are the capabilities that a centralized and holistic view of a platform’s metadata can enable?
How would you characterize the current state and progress on the open source initiative around OpenMetadata?
How does OpenMetadata compare to the OpenLineage project and other similar systems?
What opportunities do you see for collaborating with or learning from their efforts?
What are the schema elements that you have identified as critical to a holistic view of an organization’s metadata?
For an organization with an existing data platform, what is the role that OpenMetadata plays, and what are the points of integration across the different components?
Can you describe the implementation of the OpenMetadata architecture?
What are the user experience and operational characteristics that you are trying to optimize for as you iterate on the project?
What are the challenges that you face in balancing the generality and specificity of the core schemas for metadata objects?
There are a large and growing number of businesses that create systems on top of an organizations metadata in the form of catalogs, observability, governance, data quality, etc. What do you see as the role of the OpenMetadata project across that ecosystem of products?
How has your perspective on the domain of metadata management and the associated challenges changed or evolved as you have been working on this project?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on OpenMetadata?
When is OpenMetadata the wrong choice?
What do you have planned for the future of OpenMetadata?
Contact Info
Suresh
LinkedIn
@suresh_m_s on Twitter
sureshms on GitHub
Sriharsha
LinkedIn
harshach on GitHub
@d3fmacro on Twitter
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
OpenMetadata
Apache Storm
Apache Kafka
Hortonworks
Apache Atlas
OpenMetadata Sandbox
OpenLineage
Podcast Episode
Egeria
JSON Schema
Amundsen
Podcast Episode
DataHub
Podcast Episode
JanusGraph
Titan Graph Database
HBase
Jetty
DropWizard
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
1 hr 7 min
November 06, 2021Business Intelligence Beyond The Dashboard With ClicData
Summary
Business intelligence is often equated with a collection of dashboards that show various charts and graphs representing data for an organization. What is overlooked in that characterization is the level of complexity and effort that are required to collect and present that information, and the opportunities for providing those insights in other contexts. In this episode Telmo Silva explains how he co-founded ClicData to bring full featured business intelligence and reporting to every organization without having to build and maintain that capability on their own. This is a great conversation about the technical and organizational operations involved in building a comprehensive business intelligence system and the current state of the market.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days. Datafold helps Data teams gain visibility and confidence in the quality of their analytical data through data profiling, column-level lineage and intelligent anomaly detection. Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Go to dataengineeringpodcast.com/datafold today to start a 30-day trial of Datafold.
Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription
Your host is Tobias Macey and today I’m interviewing Telmo Silva about ClicData,
Interview
Introduction
How did you get involved in the area of data management?
Can you describe what ClicData is and the story behind it?
How would you characterize the current state of the market for business intelligence?
What are the systems/capabilities that are required to run a full-featured BI system?
What are the challenges that businesses face in developing in-house capacity for business intelligence?
Can you describe how the ClicData platform is architected?
How has it changed or evolved since you first began working on it?
How are you approaching schema design and evolution in the storage layer?
How do you handle questions of data security/privacy/regulations given that you are storing the information on behalf of the business?
In your work with clients what are some of the challenges that businesses are facing when attempting to answer questions and gain insights from their data in a repeatable fashion?
What are some strategies that you have found useful for structuring schemas or dashboards to make iterative exploration of data effective?
What are the most interesting, innovative, or unexpected ways that you have seen ClicData used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on ClicData?
When is ClicData the wrong choice?
What do you have planned for the future of ClicData?
Contact Info
LinkedIn
@telmo_clicdata on Twitter
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
ClicData
Tableau
Superset
Podcast Episode
Pentaho
D3.js
Informatica
Talend
TIBCO Spotfire
Looker
Podcast Episode
Bullet Chart
PostgreSQL
Podcast Episode
Azure
Crystal Reports
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
1 hr 2 min
November 05, 2021Exploring The Evolution And Adoption of Customer Data Platforms and Reverse ETL
Summary
The precursor to widespread adoption of cloud data warehouses was the creation of customer data platforms. Acting as a centralized repository of information about how your customers interact with your organization they drove a wave of analytics about how to improve products based on actual usage data. A natural outgrowth of that capability is the more recent growth of reverse ETL systems that use those analytics to feed back into the operational systems used to engage with the customer. In this episode Tejas Manohar and Rachel Bradley-Haas share the story of their own careers and experiences coinciding with these trends. They also discuss the current state of the market for these technological patterns and how to take advantage of them in your own work.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Struggling with broken pipelines? Stale dashboards? Missing data? If this resonates with you, you’re not alone. Data engineers struggling with unreliable data need look no further than Monte Carlo, the world’s first end-to-end, fully automated Data Observability Platform! In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Monte Carlo monitors and alerts for data issues across your data warehouses, data lakes, ETL, and business intelligence, reducing time to detection and resolution from weeks or days to just minutes. Go to dataengineeringpodcast.com/montecarlo and start trusting your data with Monte Carlo today!
Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads? Hightouch is the easiest way to sync data into the platforms that your business teams rely on. The data you’re looking for is already in your data warehouse and BI tools. Connect your warehouse to Hightouch, paste a SQL query, and use their visual mapper to specify how data should appear in your SaaS systems. No more scripts, just SQL. Supercharge your business teams with customer data using Hightouch for Reverse ETL today. Get started for free at dataengineeringpodcast.com/hightouch.
Your host is Tobias Macey and today I’m interviewing Rachel Bradley-Haas and Tejas Manohar about the combination of operational analytics and the customer data platform
Interview
Introduction
How did you get involved in the area of data management?
Can we start by discussing what it means to have a "customer data platform"?
What are the challenges that organizations face in establishing a unified view of their customer interactions?
How do the presence of multiple product lines impact the ability to understand the relationship with the customer?
We have been building data warehouses and business intelligence systems for decades. How does the idea of a CDP differ from the approaches of those previous generations?
A recent outgrowth of the focus on creating a CDP is the introduction of "operational analytics", which was initially termed "reverse ETL". What are your opinions on the semantics and importance of these names?
What is the relationship between a CDP and operational analytics? (can you have one without the other?)
How have the capabilities of operational analytics systems changed or evolved in the past couple of years?
What new use cases or capabilities have been unlocked as a result of these changes?
What are the opportunities over the medium to long term for operational analytics and customer data platforms?
What are the most interesting, innovative, or unexpected ways that you have seen operational analytics and CDPs used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on operational analytics?
When is a CDP the wrong choice?
What other industry trends are you keeping an eye on? What do you anticipate will be the next breakout product category?
Contact Info
Rachel
LinkedIn
Tejas
LinkedIn
@tejasmanohar on Twitter
tejasmanohar on GitHub
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
Big-Time Data
Hightouch
Podcast Episode
Segment
Podcast Episode
Customer Data Platform
Treasure Data
Rudderstack
Airflow
DBT Cloud
Fivetran
Podcast Episode
Stitch
PLG == Product Led Growth
ABM == Account Based Marketing
Materialize
Podcast Episode
Transform
Podcast Episode
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
1 hr 3 min
October 29, 2021Removing The Barrier To Exploratory Analytics with Activity Schema and Narrator
Summary
The perennial question of data warehousing is how to model the information that you are storing. This has given rise to methods as varied as star and snowflake schemas, data vault modeling, and wide tables. The challenge with many of those approaches is that they are optimized for answering known questions but brittle and cumbersome when exploring unknowns. In this episode Ahmed Elsamadisi shares his journey to find a more flexible and universal data model in the form of the "activity schema" that is powering the Narrator platform, and how it has allowed his customers to perform self-service exploration of their business domains without being blocked by schema evolution in the data warehouse. This is a fascinating exploration of what can be done when you challenge your assumptions about what is possible.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription
Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days. Datafold helps Data teams gain visibility and confidence in the quality of their analytical data through data profiling, column-level lineage and intelligent anomaly detection. Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Go to dataengineeringpodcast.com/datafold today to start a 30-day trial of Datafold.
Your host is Tobias Macey and today I’m interviewing Ahmed Elsamadisi about Narrator, a platform to enable anyone to go from question to data-driven decision in minutes
Interview
Introduction
How did you get involved in the area of data management?
Can you describe what Narrator is and the story behind it?
What are the challenges that you have seen organizations encounter when attempting to make analytics a self-serve capability?
What are the use cases that you are focused on?
How does Narrator fit within the data workflows of an organization?
How is the Narrator platform implemented?
How has the design and focus of the technology evolved since you first started working on Narrator?
The core element of the analyses that you are building is the "activity schema". Can you describe the design process that led you to that format?
What are the challenges that are posed by more widely used modeling techniques such as star/snowflake or data vault?
How does the activity schema address those challenges?
What are the performance characteristics of deriving models from an activity schema/timeseries table?
For someone who wants to use Narrator, what is involved in transforming their data to map into the activity schema?
Can you talk through the domain modeling that needs to happen when determining what entities and actions to capture?
What are the most interesting, innovative, or unexpected ways that you have seen Narrator used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Narrator?
When is Narrator the wrong choice?
What do you have planned for the future of Narrator?
Contact Info
LinkedIn
@ae4ai on Twitter
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
Narrator
DARPA Challenge
Fivetran
Luigi
Chartio
Airflow
Domain Driven Design
Data Vault
Snowflake Schema
Event Sourcing
Census
Podcast Episode
Hightouch
Podcast Episode
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
1 hr 9 min
October 29, 2021Streaming Data Pipelines Made SQL With Decodable
Summary
Streaming data systems have been growing more capable and flexible over the past few years. Despite this, it is still challenging to build reliable pipelines for stream processing. In this episode Eric Sammer discusses the shortcomings of the current set of streaming engines and how they force engineers to work at an extremely low level of abstraction. He also explains why he started Decodable to address that limitation and the work that he and his team have done to let data engineers build streaming pipelines entirely in SQL.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Struggling with broken pipelines? Stale dashboards? Missing data? If this resonates with you, you’re not alone. Data engineers struggling with unreliable data need look no further than Monte Carlo, the world’s first end-to-end, fully automated Data Observability Platform! In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Monte Carlo monitors and alerts for data issues across your data warehouses, data lakes, ETL, and business intelligence, reducing time to detection and resolution from weeks or days to just minutes. Start trusting your data with Monte Carlo today! Visit dataengineeringpodcast.com/impact today to save your spot at IMPACT: The Data Observability Summit a half-day virtual event featuring the first U.S. Chief Data Scientist, founder of the Data Mesh, Creator of Apache Airflow, and more data pioneers spearheading some of the biggest movements in data. The first 50 to RSVP with this link will be entered to win an Oculus Quest 2 — Advanced All-In-One Virtual Reality Headset. RSVP today – you don’t want to miss it!
Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads? Hightouch is the easiest way to sync data into the platforms that your business teams rely on. The data you’re looking for is already in your data warehouse and BI tools. Connect your warehouse to Hightouch, paste a SQL query, and use their visual mapper to specify how data should appear in your SaaS systems. No more scripts, just SQL. Supercharge your business teams with customer data using Hightouch for Reverse ETL today. Get started for free at dataengineeringpodcast.com/hightouch.
Your host is Tobias Macey and today I’m interviewing Eric Sammer about Decodable, a platform for simplifying the work of building real-time data pipelines
Interview
Introduction
How did you get involved in the area of data management?
Can you describe what Decodable is and the story behind it?
Who are the target users, and how has that focus informed your prioritization of features at launch?
What are the complexities that data engineers encounter when building pipelines on streaming systems?
What are the distributed systems concepts and design optimizations that are often skipped over or misunderstood by engineers who are using them? (e.g. backpressure, exactly once semantics, isolation levels, etc.)
How do those mismatches in understanding and expectation impact the correctness and reliability of the workflows that they are building?
Can you describe how you have architected the Decodable platform?
What have been the most complex or time consuming engineering challenges that you have dealt with so far?
What are the points of integration that you expose for engineers to wire in their existing infrastructure and data systems?
What has been your process for designing the interfaces and abstractions that you are exposing to end users?
What are some of the leaks in those abstractions that have either started to show or are anticipated?
What have you learned about the state of data engineering and the costs and benefits of real-time data while working on Decodable?
What are the most interesting, innovative, or unexpected ways that you have seen Decodable used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Decodable?
When is Decodable the wrong choice?
What do you have planned for the future of Decodable?
Contact Info
esammer on GitHub
@esammer on Twitter
LinkedIn
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
Decodable
Cloudera
Kafka
Flink
Podcast Episode
Spark
Snowflake
Podcast Episode
BigQuery
RedShift
kSQLDB
Podcast Episode
dbt
Podcast Episode
Millwheel Paper
Dremel Paper
Timely Dataflow
Materialize
Podcast Episode
Software Defined Networking
Data Mesh
Podcast Episode
OpenLineage
Podcast Episode
DataHub
Podcast Episode
Amundsen
Podcast Episode
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
1 hr 10 min
October 23, 2021Data Exploration For Business Users Powered By Analytics Engineering With Lightdash
Summary
The market for business intelligence has been going through an evolutionary shift in recent years. One of the driving forces for that change has been the rise of analytics engineering powered by dbt. Lightdash has fully embraced that shift by building an entire open source business intelligence framework that is powered by dbt models. In this episode Oliver Laslett describes why dashboards aren’t sufficient for business analytics, how Lightdash promotes the work that you are already doing in your data warehouse modeling with dbt, and how they are focusing on bridging the divide between data teams and business teams and the requirements that they have for data workflows.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads? Hightouch is the easiest way to sync data into the platforms that your business teams rely on. The data you’re looking for is already in your data warehouse and BI tools. Connect your warehouse to Hightouch, paste a SQL query, and use their visual mapper to specify how data should appear in your SaaS systems. No more scripts, just SQL. Supercharge your business teams with customer data using Hightouch for Reverse ETL today. Get started for free at dataengineeringpodcast.com/hightouch.
Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days. Datafold helps Data teams gain visibility and confidence in the quality of their analytical data through data profiling, column-level lineage and intelligent anomaly detection. Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Go to dataengineeringpodcast.com/datafold today to start a 30-day trial of Datafold.
Your host is Tobias Macey and today I’m interviewing Oliver Laslett about Lightdash, an open source business intelligence system powered by your dbt models
Interview
Introduction
How did you get involved in the area of data management?
Can you describe what Lightdash is and the story behind it?
What are the main goals of the project?
Who are the target users, and how has that profile informed your feature priorities?
Business intelligence is a market that has gone through several generational shifts, with products targeting numerous personas and purposes. What are the capabilities that make Lightdash stand out from the other options?
Can you describe how Lightdash is architected?
How have the design and goals of the system changed or evolved since you first began working on it?
What have been the most challenging engineering problems that you have dealt with?
How does the approach that you are taking with Lightdash compare to systems such as Transform and Metriql that aim to provide a dedicated metrics layer?
Can you describe the workflow for someone building an analysis in Lightdash?
What are the points of collaboration around Lightdash for different roles in the organization?
What are the methods that you use to expose information about the state of the underlying dbt models to the end users?
How do they use that information in their exploration and decision making?
What was your motivation for releasing Lightdash as open source?
How are you handling the governance and long-term viability of the project?
What are the most interesting, innovative, or unexpected ways that you have seen Lightdash used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Lightdash?
When is Lightdash the wrong choice?
What do you have planned for the future of Lightdash?
Contact Info
LinkedIn
owlas on GitHub
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
Lightdash
Looker
Podcast Episode
PowerBI
Podcast Episode
Redash
Podcast Episode
Metabase
Podcast Episode
dbt
Podcast Episode
Superset
Podcast Episode
Streamlit
Podcast Episode
Kubernetes
JDBC
SQLAlchemy
SQLPad
Singer
Podcast Episode
Airbyte
Podcast Episode
Meltano
Podcast Episode
Transform
Podcast Episode
Metriql
Podcast Episode
Cube.js
OpenLineage
Podcast Episode
dbt Packages
Rudderstack
PostHog
Podcast Interview
Firebolt
Podcast Interview
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
1 hr 7 min
October 21, 2021Completing The Feedback Loop Of Data Through Operational Analytics With Census
Summary
The focus of the past few years has been to consolidate all of the organization’s data into a cloud data warehouse. As a result there have been a number of trends in data that take advantage of the warehouse as a single focal point. Among those trends is the advent of operational analytics, which completes the cycle of data from collection, through analysis, to driving further action. In this episode Boris Jabes, CEO of Census, explains how the work of synchronizing cleaned and consolidated data about your customers back into the systems that you use to interact with those customers allows for a powerful feedback loop that has been missing in data systems until now. He also discusses how Census makes that synchronization easy to manage, how it fits with the growth of data quality tooling, and how you can start using it today.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Struggling with broken pipelines? Stale dashboards? Missing data? If this resonates with you, you’re not alone. Data engineers struggling with unreliable data need look no further than Monte Carlo, the world’s first end-to-end, fully automated Data Observability Platform! In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Monte Carlo monitors and alerts for data issues across your data warehouses, data lakes, ETL, and business intelligence, reducing time to detection and resolution from weeks or days to just minutes. Start trusting your data with Monte Carlo today! Visit dataengineeringpodcast.com/impact today to save your spot at IMPACT: The Data Observability Summit a half-day virtual event featuring the first U.S. Chief Data Scientist, founder of the Data Mesh, Creator of Apache Airflow, and more data pioneers spearheading some of the biggest movements in data. The first 50 to RSVP with this link will be entered to win an Oculus Quest 2 — Advanced All-In-One Virtual Reality Headset. RSVP today – you don’t want to miss it!
Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription
Your host is Tobias Macey and today I’m interviewing Boris Jabes about Census and the growing category of operational analytics
Interview
Introduction
How did you get involved in the area of data management?
Can you describe what Census is and the story behind it?
The terms "reverse ETL" and "operational analytics" have started being used for similar, and often interchangeable, purposes. What are your thoughts on the semantic and concrete differences between these phrases?
What are the motivating factors for adding operational analytics or "data activation" to an organization’s data platform?
This is a nascent but quickly growing market with a number of products and projects operating in the space. How would you characterize the current state of the segment and Census’ position in it?
Can you describe how the Census platform is implemented?
What are some of the early design choices that have had to be refactored or augmented as you have evolved the product and worked with customers?
What are some of the assumptions that you had about the needs and uses for the platform which have been challenged or changed as you dug deeper into the problem?
Can you describe the workflow for a customer adopting Census?
What are some of the data modeling practices that make it easier to "activate" the organization’s data?
Another recent trend in the data industry is the growth of data quality and data lineage tools. What is involved in using the measured quality or lineage information as a signal in the operational systems, or to prevent a synchronization?
How can users test and validate their workflows in Census?
What are the options for propagating Census’ runtime information back into lineage and data quality tracking?
Census supports incremental syncs from the warehouse. What are the opportunities for bringing streaming architectures to the space of operational analytics?
What are the challenges/complexities in the current set of technologies that act as a barrier?
What are the most interesting, innovative, or unexpected ways that you have seen Census used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Census?
When is Census the wrong choice?
What do you have planned for the future of Census?
Contact Info
LinkedIn
Website
@borisjabes on Twitter
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
Census
Operational Analytics
Fivetran
Podcast Episode
dbt
Podcast Episode
Snowflake
Podcast Episode
Loom
Materialize
Podcast Episode
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
1 hr 10 min
October 16, 2021Bringing The Power Of The DataHub Real-Time Metadata Graph To Everyone At Acryl Data
Summary
The binding element of all data work is the metadata graph that is generated by all of the workflows that produce the assets used by teams across the organization. The DataHub project was created as a way to bring order to the scale of LinkedIn’s data needs. It was also designed to be able to work for small scale systems that are just starting to develop in complexity. In order to support the project and make it even easier to use for organizations of every size Shirshanka Das and Swaroop Jagadish founded Acryl Data. In this episode they discuss the recent work that has been done by the community, how their work is building on top of that foundation, and how you can get started with DataHub for your own work to manage data discovery today. They also share their ambitions for the near future of adding data observability and data quality management features.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads? Hightouch is the easiest way to sync data into the platforms that your business teams rely on. The data you’re looking for is already in your data warehouse and BI tools. Connect your warehouse to Hightouch, paste a SQL query, and use their visual mapper to specify how data should appear in your SaaS systems. No more scripts, just SQL. Supercharge your business teams with customer data using Hightouch for Reverse ETL today. Get started for free at dataengineeringpodcast.com/hightouch.
Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days. Datafold helps Data teams gain visibility and confidence in the quality of their analytical data through data profiling, column-level lineage and intelligent anomaly detection. Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Go to dataengineeringpodcast.com/datafold today to start a 30-day trial of Datafold. Once you sign up and create an alert in Datafold for your company data, they will send you a cool water flask.
Your host is Tobias Macey and today I’m interviewing Shirshanka Das and Swaroop Jagadish about Acryl Data, the company driving the open source metadata project DataHub for powering data discovery, data observability and federated data governance.
Interview
Introduction
How did you get involved in the area of data management?
Can you describe what Acryl Data is and the story behind it?
How has your experience of building and running DataHub at LinkedIn informed your product direction for Acryl?
What are some lessons that your co-founder Swaroop has contributed from his experience at AirBnB?
The data catalog/discovery/quality market has become very active over the past year. What is your perspective on the market, and what are the gaps that are not yet being addressed?
How does the focus of Acryl compare to what the team at Metaphor are building?
How has the DataHub project changed in the past year with more companies outside of LinkedIn using and contributing to it?
What are your plans for Data Observability?
Can you describe the system architecture that you have built at Acryl?
What are the convenience features that you are building to augment the capabilities and integration process for DataHub?
What are some typical workflows that data teams build out when working with Acryl?
What are some examples of automated actions that can be triggered from metadata changes?
What are the available events that can be used to trigger actions?
What are some of the challenges that teams are facing when integrating metadata management and analysis into their data workflows?
What are your thoughts on the potential for the Open Lineage and Open metadata projects?
How is the governance of DataHub being managed?
What are the most interesting, innovative, or unexpected ways that you have seen Acryl/DataHub used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Acryl/DataHub?
When is Acryl the wrong choice?
What do you have planned for the future of Acryl?
Contact Info
Shirshanka
LinkedIn
@shirshanka on Twitter
shirshanka on GitHub
Swaroop
LinkedIn
@arudis on Twitter
swaroopjagadish on GitHub
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
Acryl Data
DataHub
Hudi
Podcast Episode
Iceberg
Podcast Episode
Delta Lake
Podcast Episode
Apache Gobblin
Airflow
Superset
Podcast Episode
Collibra
Podcast Episode
Alation
Strata Conference Presentation
Acryl/DataHub Ingestion Framework
Joe Hellerstein
Trifacta
DataHub Roadmap
Data Mesh
OpenLineage
Podcast Episode
OpenMetadata
Egeria Open Metadata
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
1 hr 9 min
October 14, 2021How And Why To Become Data Driven As A Business
Summary
Organizations of all sizes are striving to become data driven, starting in earnest with the rise of big data a decade ago. With the never-ending growth in data sources and methods for aggregating and analyzing them, the use of data to direct the business has become a requirement. Randy Bean has been helping enterprise organizations define and execute their data strategies since before the age of big data. In this episode he discusses his experiences and how he approached the work of distilling them for his book "Fail Fast, Learn Faster". This is an entertaining and enlightening exploration of the business side of data with an industry veteran.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Struggling with broken pipelines? Stale dashboards? Missing data? If this resonates with you, you’re not alone. Data engineers struggling with unreliable data need look no further than Monte Carlo, the world’s first end-to-end, fully automated Data Observability Platform! In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Monte Carlo monitors and alerts for data issues across your data warehouses, data lakes, ETL, and business intelligence, reducing time to detection and resolution from weeks or days to just minutes. Start trusting your data with Monte Carlo today! Visit dataengineeringpodcast.com/impact today to save your spot at IMPACT: The Data Observability Summit a half-day virtual event featuring the first U.S. Chief Data Scientist, founder of the Data Mesh, Creator of Apache Airflow, and more data pioneers spearheading some of the biggest movements in data. The first 50 to RSVP with this link will be entered to win an Oculus Quest 2 — Advanced All-In-One Virtual Reality Headset. RSVP today – you don’t want to miss it!
Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription
Your host is Tobias Macey and today I’m interviewing Randy Bean about his recent book focusing on the use of big data and AI for informing data driven business leadership
Interview
Introduction
How did you get involved in the area of data management?
Can you start by discussing the focus of the book and what motivated you to write it?
Who is the intended audience, and how did that inform the tone and content?
Businesses and their officers have been aiming to be "data driven" for years. In your experience, what are the concrete goals that are implied by that term?
What are the barriers that organizations encounter in the pursuit of those goals?
How have the success rates (real and imagined) shifted in recent years as the level of sophistication of the tools and industry for data management has increased?
What is the state of data initiatives in leading corporations today?
What are the biggest opportunities and risks that organizations focus on related to their use of data?
At what level(s) of the organization do lessons around data ethics need to be embedded?
You have been working with large companies for many years to help them with their adoption of "big data". How has your work on this book shifted or clarified your perspectives on the subject?
What are the main lessons or ideas that you hope readers will take away from the book?
What are the most interesting, innovative, or unexpected ways that you have seen big data applied to business?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on this book?
What are your predictions for the next decade of big data and AI?
Contact Info
@RandyBeanNVP on Twitter
LinkedIn
Email
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
Fail Fast, Learn Faster: Lessons in Data-Driven Leadership in an Age of Disruption, Big Data, and AI (affiliate link)
Book Website
Harvard Business Review
MIT Sloan Review
New Vantage Partners
COBOL
Moneyball
Weapons of Math Destruction
The Seven Roles of the Chief Data Officer
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
1 hr 2 min
October 08, 2021Make Your Business Metrics Reusable With Open Source Headless BI Using Metriql
Summary
The key to making data valuable to business users is the ability to calculate meaningful metrics and explore them along useful dimensions. Business intelligence tools have provided this capability for years, but they don’t offer a means of exposing those metrics to other systems. Metriql is an open source project that provides a headless BI system where you can define your metrics and share them with all of your other processes. In this episode Burak Kabakcı shares the story behind the project, how you can use it to create your metrics definitions, and the benefits of treating the semantic layer as a dedicated component of your platform.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription
Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days. Datafold helps Data teams gain visibility and confidence in the quality of their analytical data through data profiling, column-level lineage and intelligent anomaly detection. Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Go to dataengineeringpodcast.com/datafold today to start a 30-day trial of Datafold. Once you sign up and create an alert in Datafold for your company data, they will send you a cool water flask.
Your host is Tobias Macey and today I’m interviewing Burak Emre Kabakcı about Metriql, a headless BI and metrics layer for your data stack
Interview
Introduction
How did you get involved in the area of data management?
Can you describe what Metriql is and the story behind it?
What are the characteristics and benefits of a "headless BI" system?
What was your motivation to create and open-source Metriql as an independent project outside of your business?
How are you approaching governance and sustainability of the project?
How does Metriql compare to projects such as AirBnB’s Minerva or Transform’s platform?
How does the industry/vertical of a business impact their ability to benefit from a metrics layer/headless BI?
What are the limitations to the logical complexity that can be applied to the calculation of a given metric/set of metrics?
Can you describe how Metriql is implemented?
How have the design and goals of the project changed or evolved since you began working on it?
What are the most complex/difficult engineering elements of building a metrics layer?
Can you describe the workflow of defining metrics?
What have been your guiding principles in defining the user experience for working with metriql?
What are the opportunities for including business users in the definition of metrics? (e.g. pushing down/generating definitions from a BI layer)
What are the biggest challenges and limitations of creating metrics definitions purely in SQL?
What are the options for exposing metrics back to the warehouse and other operational systems such as reverse ETL vendors?
What are the missing elements in the data ecosystem for taking full advantage of a headless BI/metrics layer?
What are the most interesting, innovative, or unexpected ways that you have seen Metriql used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Metriql?
When is Metriql the wrong choice?
What do you have planned for the future of Metriql?
Contact Info
LinkedIn
Website
buremba on GitHub
@bu7emba on Twitter
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
Metriql
Rakam
Hazelcast
Headless BI
Google Data Studio
Superset
Podcast Episode
Podcast.__init__ Episode
Trino
Podcast Episode
Supergrain
The Missing Piece Of The Modern Data Stack article by Benn Stancil
Metabase
Podcast Episode
dbt
Podcast Episode
dbt-metabase
re_data
OpenMetadata
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
44 min

About Data Engineering Podcast

From the publisher's feed

This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some…