Data Engineering Podcast

By Tobias Macey

What's Data Engineering Podcast about?

This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.

Download our free app to listen on your phone

Data Engineering Podcast episodes:

04.21.2024
Making Email Better With AI At Shortwave
Summary Generative AI has rapidly transformed everything in the technology sector. When Andrew Lee started work on Shortwave he was focused on making email more productive. When AI started gaining adoption he realized that he had even more potential for a...
04.14.2024
Designing A Non-Relational Database Engine
Summary Databases come in a variety of formats for different use cases. The default association with the term "database" is relational engines, but non-relational engines are also used quite widely. In this episode Oren Eini, CEO and creator of RavenDB, explores...
04.07.2024
Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer
Summary Maintaining a single source of truth for your data is the biggest challenge in data engineering. Different roles and tasks in the business need their own ways to access and analyze the data in the organization. In order to enable...
03.31.2024
Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary
Summary Working with data is a complicated process, with numerous chances for something to go wrong. Identifying and accounting for those errors is a critical piece of building trust in the organization that your data is accurate and up to date....
03.24.2024
Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+
Summary A core differentiator of Dagster in the ecosystem of data orchestration is their focus on software defined assets as a means of building declarative workflows. With their launch of Dagster+ as the redesigned commercial companion to the open source project...
03.17.2024
Reconciling The Data In Your Databases With Datafold
Summary A significant portion of data workflows involve storing and processing information in database engines. Validating that the information is stored and processed correctly can be complex and time-consuming, especially when the source and destination speak different dialects of SQL. In...
03.10.2024
Version Your Data Lakehouse Like Your Software With Nessie
Summary Data lakehouse architectures are gaining popularity due to the flexibility and cost effectiveness that they offer. The link that bridges the gap between data lake and warehouse capabilities is the catalog. The primary purpose of the catalog is to inform...
03.03.2024
When And How To Conduct An AI Program
Summary Artificial intelligence technologies promise to revolutionize business and produce new sources of value. In order to make those promises a reality there is a substantial amount of strategy and investment required. Colleen Tartow has worked across all stages of the...
02.25.2024
Find Out About The Technology Behind The Latest PFAD In Analytical Database Development
Summary Building a database engine requires a substantial amount of engineering effort and time investment. Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. When Paul...
02.18.2024
Using Trino And Iceberg As The Foundation Of Your Data Lakehouse
Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Multiple open source projects and vendors have been working together to make this vision a...
02.11.2024
Data Sharing Across Business And Platform Boundaries
Summary Sharing data is a simple concept, but complicated to implement well. There are numerous business rules and regulatory concerns that need to be applied. There are also numerous technical considerations to be made, particularly if the producer and consumer of...
02.04.2024
Tackling Real Time Streaming Data With SQL Using RisingWave
Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer....
01.29.2024
Build A Data Lake For Your Security Logs With Scanner
Summary Monitoring and auditing IT systems for security events requires the ability to quickly analyze massive volumes of unstructured log data. The majority of products that are available either require too much effort to structure the logs, or aren't fast enough...
01.22.2024
Modern Customer Data Platform Principles
Summary Databases and analytics architectures have gone through several generational shifts. A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization. In this episode Tasso Argyros, CEO of...
01.07.2024
Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel
Summary Data processing technologies have dramatically improved in their sophistication and raw throughput. Unfortunately, the volumes of data that are being generated continue to double, requiring further advancements in the platform capabilities to keep up. As the sophistication increases, so does...
01.01.2024
Designing Data Platforms For Fintech Companies
Summary Working with financial data requires a high degree of rigor due to the numerous regulations and the risks involved in security breaches. In this episode Andrey Korchack, CTO of fintech startup Monite, discusses the complexities of designing and implementing a...
12.24.2023
Troubleshooting Kafka In Production
Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Operating it at scale, however, is notoriously challenging. Elad Eldor has experienced these challenges first-hand, leading to his work writing the book...
12.18.2023
Adding An Easy Mode For The Modern Data Stack With 5X
Summary The "modern data stack" promised a scalable, composable data platform that gave everyone the flexibility to use the best tools for every job. The reality was that it left data teams in the position of spending all of their engineering...
12.11.2023
Run Your Own Anomaly Detection For Your Critical Business Metrics With Anomstack
Summary If your business metrics looked weird tomorrow, would you know about it first? Anomaly detection is focused on identifying those outliers for you, so that you are the first to know when a business critical dashboard isn't right. Unfortunately, it...
12.04.2023
Designing Data Transfer Systems That Scale
Summary The first step of data pipelines is to move the data to a place where you can process and prepare it for its eventual purpose. Data transfer systems are a critical component of data enablement, and building them to support...
11.27.2023
Addressing The Challenges Of Component Integration In Data Platform Architectures
Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. One of the core complexities that needs to be addressed is the fractal set of integrations that need to be managed...
11.20.2023
Unlocking Your dbt Projects With Practical Advice For Practitioners
Summary The dbt project has become overwhelmingly popular across analytics and data engineering teams. While it is easy to adopt, there are many potential pitfalls. Dustin Dorsey and Cameron Cyr co-authored a practical guide to building your dbt project. In this...
11.13.2023
Enhancing The Abilities Of Software Engineers With Generative AI At Tabnine
Summary Software development involves an interesting balance of creativity and repetition of patterns. Generative AI has accelerated the ability of developer tools to provide useful suggestions that speed up the work of engineers. Tabnine is one of the main platforms offering...
11.06.2023
Shining Some Light In The Black Box Of PostgreSQL Performance
Summary Databases are the core of most applications, but they are often treated as inscrutable black boxes. When an application is slow, there is a good probability that the database needs some attention. In this episode Lukas Fittl shares some hard-won...
10.30.2023
Surveying The Market Of Database Products
Summary Databases are the core of most applications, whether transactional or analytical. In recent years the selection of database products has exploded, making the critical decision of which engine(s) to use even more difficult. In this episode Tanya Bragin shares her...
10.23.2023
Defining A Strategy For Your Data Products
Summary The primary application of data has moved beyond analytics. With the broader audience comes the need to present data in a more approachable format. This has led to the broad adoption of data products being the delivery mechanism for information....
10.15.2023
Reducing The Barrier To Entry For Building Stream Processing Applications With Decodable
Summary Building streaming applications has gotten substantially easier over the past several years. Despite this, it is still operationally challenging to deploy and maintain your own stream processing infrastructure. Decodable was built with a mission of eliminating all of the painful...
10.09.2023
Using Data To Illuminate The Intentionally Opaque Insurance Industry
Summary The insurance industry is notoriously opaque and hard to navigate. Max Cho found that fact frustrating enough that he decided to build a business of making policy selection more navigable. In this episode he shares his journey of data collection...
10.01.2023
Building ETL Pipelines With Generative AI
Summary Artificial intelligence applications require substantial high quality data, which is provided through ETL pipelines. Now that AI has reached the level of sophistication seen in the various generative models it is being used to build new ETL workflows. In this...
09.25.2023
Powering Vector Search With Real Time And Incremental Vector Indexes
Summary The rapid growth of machine learning, especially large language models, have led to a commensurate growth in the need to store and compare vectors. In this episode Louis Brandy discusses the applications for vector search capabilities both in and outside...
09.17.2023
Building Linked Data Products With JSON-LD
Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. Linked data technologies provide a means of tightly coupling metadata with raw information. In this episode Brian Platz explains how...
09.10.2023
An Overview Of The State Of Data Orchestration In An Increasingly Complex Data Ecosystem
Summary Data systems are inherently complex and often require integration of multiple technologies. Orchestrators are centralized utilities that control the execution and sequencing of interdependent operations. This offers a single location for managing visibility and error handling so that data platform...
09.04.2023
Eliminate The Overhead In Your Data Integration With The Open Source dlt Library
Summary Cloud data warehouses and the introduction of the ELT paradigm has led to the creation of multiple options for flexible data integration, with a roughly equal distribution of commercial and open source options. The challenge is that most of those...
08.28.2023
Building An Internal Database As A Service Platform At Cloudflare
Summary Data persistence is one of the most challenging aspects of computer systems. In the era of the cloud most developers rely on hosted services to manage their databases, but what if you are a cloud service? In this episode Vignesh...
08.20.2023
Harnessing Generative AI For Creating Educational Content With Illumidesk
Summary Generative AI has unlocked a massive opportunity for content creation. There is also an unfulfilled need for experts to be able to share their knowledge and build communities. Illumidesk was built to take advantage of this intersection. In this episode...
08.14.2023
Unpacking The Seven Principles Of Modern Data Pipelines
Summary Data pipelines are the core of every data product, ML model, and business intelligence dashboard. If you're not careful you will end up spending all of your time on maintenance and fire-fighting. The folks at Rivery distilled the seven principles...
08.06.2023
Quantifying The Return On Investment For Your Data Team
Summary As businesses increasingly invest in technology and talent focused on data engineering and analytics, they want to know whether they are benefiting. So how do you calculate the return on investment for data? In this episode Barr Moses and Anna...
07.31.2023
Strategies For A Successful Data Platform Migration
Summary All software systems are in a constant state of evolution. This makes it impossible to select a truly future-proof technology stack for your data platform, making an eventual migration inevitable. In this episode Gleb Mezhanskiy and Rob Goretsky share their...
07.24.2023
Build Real Time Applications With Operational Simplicity Using Dozer
Summary Real-time data processing has steadily been gaining adoption due to advances in the accessibility of the technologies involved. Despite that, it is still a complex set of capabilities. To bring streaming data in reach of application engineers Matteo Pelati helped...
07.17.2023
Datapreneurs - How Todays Business Leaders Are Using Data To Define The Future
Summary Data has been one of the most substantial drivers of business and economic value for the past few decades. Bob Muglia has had a front-row seat to many of the major shifts driven by technology over his career. In his...
07.09.2023
Reduce Friction In Your Business Analytics Through Entity Centric Data Modeling
Summary For business analytics the way that you model the data in your warehouse has a lasting impact on what types of questions can be answered quickly and easily. The major strategies in use today were created decades ago when the...
07.03.2023
How Data Engineering Teams Power Machine Learning With Feature Platforms
Summary Feature engineering is a crucial aspect of the machine learning workflow. To make that possible, there are a number of technical and procedural capabilities that must be in place first. In this episode Razi Raziuddin shares how data engineering teams...
06.25.2023
Seamless SQL And Python Transformations For Data Engineers And Analysts With SQLMesh
Summary Data transformation is a key activity for all of the organizational roles that interact with data. Because of its importance and outsized impact on what is possible for downstream data consumers it is critical that everyone is able to collaborate...
06.18.2023
How Column-Aware Development Tooling Yields Better Data Models
Summary Architectural decisions are all based on certain constraints and a desire to optimize for different outcomes. In data systems one of the core architectural exercises is data modeling, which can have significant impacts on what is and is not possible...
06.11.2023
Build Better Tests For Your dbt Projects With Datafold And data-diff
Summary Data engineering is all about building workflows, pipelines, systems, and interfaces to provide stable and reliable data. Your data can be stable and wrong, but then it isn't reliable. Confidence in your data is achieved through constant validation and testing....
06.04.2023
Reduce The Overhead In Your Pipelines With Agile Data Engine's DataOps Service
Summary A significant portion of the time spent by data engineering teams is on managing the workflows and operations of their pipelines. DataOps has arisen as a parallel set of practices to that of DevOps teams as a means of reducing...
05.29.2023
A Roadmap To Bootstrapping The Data Team At Your Startup
Summary Building a data team is hard in any circumstance, but at a startup it can be even more challenging. The requirements are fluid, you probably don't have a lot of existing data talent to manage the hiring and onboarding, and...
05.21.2023
Keep Your Data Lake Fresh With Real Time Streams Using Estuary
Summary Batch vs. streaming is a long running debate in the world of data integration and transformation. Proponents of the streaming paradigm argue that stream processing engines can easily handle batched workloads, but the reverse isn't true. The batch world has...
05.15.2023
What Happens When The Abstractions Leak On Your Data
Summary All of the advancements in our technology is based around the principles of abstraction. These are valuable until they break down, which is an inevitable occurrence. In this episode the host Tobias Macey shares his reflections on recent experiences where...
05.07.2023
Use Consistent And Up To Date Customer Profiles To Power Your Business With Segment Unify
Summary Every business has customers, and a critical element of success is understanding who they are and how they are using the companies products or services. The challenge is that most companies have a multitude of systems that contain fragments of...

What's Data Engineering Podcast about?

Download our free app to listen on your phone

Data Engineering Podcast episodes:

Making Email Better With AI At Shortwave

Designing A Non-Relational Database Engine

Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer

Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary

Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

Reconciling The Data In Your Databases With Datafold

Version Your Data Lakehouse Like Your Software With Nessie

When And How To Conduct An AI Program

Find Out About The Technology Behind The Latest PFAD In Analytical Database Development

Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

Data Sharing Across Business And Platform Boundaries

Tackling Real Time Streaming Data With SQL Using RisingWave

Build A Data Lake For Your Security Logs With Scanner

Modern Customer Data Platform Principles

Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel

Designing Data Platforms For Fintech Companies

Troubleshooting Kafka In Production

Adding An Easy Mode For The Modern Data Stack With 5X

Run Your Own Anomaly Detection For Your Critical Business Metrics With Anomstack

Designing Data Transfer Systems That Scale

Addressing The Challenges Of Component Integration In Data Platform Architectures

Unlocking Your dbt Projects With Practical Advice For Practitioners

Enhancing The Abilities Of Software Engineers With Generative AI At Tabnine

Shining Some Light In The Black Box Of PostgreSQL Performance

Surveying The Market Of Database Products

Defining A Strategy For Your Data Products

Reducing The Barrier To Entry For Building Stream Processing Applications With Decodable

Using Data To Illuminate The Intentionally Opaque Insurance Industry

Building ETL Pipelines With Generative AI

Powering Vector Search With Real Time And Incremental Vector Indexes

Building Linked Data Products With JSON-LD

An Overview Of The State Of Data Orchestration In An Increasingly Complex Data Ecosystem

Eliminate The Overhead In Your Data Integration With The Open Source dlt Library

Building An Internal Database As A Service Platform At Cloudflare

Harnessing Generative AI For Creating Educational Content With Illumidesk

Unpacking The Seven Principles Of Modern Data Pipelines

Quantifying The Return On Investment For Your Data Team

Strategies For A Successful Data Platform Migration

Build Real Time Applications With Operational Simplicity Using Dozer

Datapreneurs - How Todays Business Leaders Are Using Data To Define The Future

Reduce Friction In Your Business Analytics Through Entity Centric Data Modeling

How Data Engineering Teams Power Machine Learning With Feature Platforms

Seamless SQL And Python Transformations For Data Engineers And Analysts With SQLMesh

How Column-Aware Development Tooling Yields Better Data Models

Build Better Tests For Your dbt Projects With Datafold And data-diff

Reduce The Overhead In Your Pipelines With Agile Data Engine's DataOps Service

A Roadmap To Bootstrapping The Data Team At Your Startup

Keep Your Data Lake Fresh With Real Time Streams Using Estuary

What Happens When The Abstractions Leak On Your Data

Use Consistent And Up To Date Customer Profiles To Power Your Business With Segment Unify