Data Engineering Podcast

By Tobias Macey

Technology

What's Data Engineering Podcast about?

This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.

Download our free app to listen on your phone

LAST EPISODE

Designing Data Transfer Sys...

12.04.2023

Summary The first step of data pipelines is to move the data to a place where you can process and prepare it for its eventual purpose. Data transfer systems are a critical component of data enablement, and building them to ...

Data Engineering Podcast episodes:

12.04.2023 Designing Data Transfer Systems That Scale Summary The first step of data pipelines is to move the data to a place where you can process and prepare it for its eventual purpose. Data transfer systems are a critical component of data enablement, and building them to ...
11.27.2023 Addressing The Challenges Of Component Integration In Data Platform Architectures Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. One of the core complexities that needs to be addressed is the fractal set of integrations that need to be ...
11.20.2023 Unlocking Your dbt Projects With Practical Advice For Practitioners Summary The dbt project has become overwhelmingly popular across analytics and data engineering teams. While it is easy to adopt, there are many potential pitfalls. Dustin Dorsey and Cameron Cyr co-authored a practical guide to building your dbt project. In ...
11.13.2023 Enhancing The Abilities Of Software Engineers With Generative AI At Tabnine Summary Software development involves an interesting balance of creativity and repetition of patterns. Generative AI has accelerated the ability of developer tools to provide useful suggestions that speed up the work of engineers. Tabnine is one of the main platforms ...
11.06.2023 Shining Some Light In The Black Box Of PostgreSQL Performance Summary Databases are the core of most applications, but they are often treated as inscrutable black boxes. When an application is slow, there is a good probability that the database needs some attention. In this episode Lukas Fittl shares some ...
10.30.2023 Surveying The Market Of Database Products Summary Databases are the core of most applications, whether transactional or analytical. In recent years the selection of database products has exploded, making the critical decision of which engine(s) to use even more difficult. In this episode Tanya Bragin shares ...
10.23.2023 Defining A Strategy For Your Data Products Summary The primary application of data has moved beyond analytics. With the broader audience comes the need to present data in a more approachable format. This has led to the broad adoption of data products being the delivery mechanism for ...
10.15.2023 Reducing The Barrier To Entry For Building Stream Processing Applications With Decodable Summary Building streaming applications has gotten substantially easier over the past several years. Despite this, it is still operationally challenging to deploy and maintain your own stream processing infrastructure. Decodable was built with a mission of eliminating all of the ...
10.09.2023 Using Data To Illuminate The Intentionally Opaque Insurance Industry Summary The insurance industry is notoriously opaque and hard to navigate. Max Cho found that fact frustrating enough that he decided to build a business of making policy selection more navigable. In this episode he shares his journey of data ...
10.01.2023 Building ETL Pipelines With Generative AI Summary Artificial intelligence applications require substantial high quality data, which is provided through ETL pipelines. Now that AI has reached the level of sophistication seen in the various generative models it is being used to build new ETL workflows. In ...
09.25.2023 Powering Vector Search With Real Time And Incremental Vector Indexes Summary The rapid growth of machine learning, especially large language models, have led to a commensurate growth in the need to store and compare vectors. In this episode Louis Brandy discusses the applications for vector search capabilities both in and ...
09.17.2023 Building Linked Data Products With JSON-LD Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. Linked data technologies provide a means of tightly coupling metadata with raw information. In this episode Brian Platz explains ...
09.10.2023 An Overview Of The State Of Data Orchestration In An Increasingly Complex Data Ecosystem Summary Data systems are inherently complex and often require integration of multiple technologies. Orchestrators are centralized utilities that control the execution and sequencing of interdependent operations. This offers a single location for managing visibility and error handling so that data ...
09.04.2023 Eliminate The Overhead In Your Data Integration With The Open Source dlt Library Summary Cloud data warehouses and the introduction of the ELT paradigm has led to the creation of multiple options for flexible data integration, with a roughly equal distribution of commercial and open source options. The challenge is that most of ...
08.28.2023 Building An Internal Database As A Service Platform At Cloudflare Summary Data persistence is one of the most challenging aspects of computer systems. In the era of the cloud most developers rely on hosted services to manage their databases, but what if you are a cloud service? In this episode ...
08.20.2023 Harnessing Generative AI For Creating Educational Content With Illumidesk Summary Generative AI has unlocked a massive opportunity for content creation. There is also an unfulfilled need for experts to be able to share their knowledge and build communities. Illumidesk was built to take advantage of this intersection. In this ...
08.14.2023 Unpacking The Seven Principles Of Modern Data Pipelines Summary Data pipelines are the core of every data product, ML model, and business intelligence dashboard. If you're not careful you will end up spending all of your time on maintenance and fire-fighting. The folks at Rivery distilled the seven ...
08.06.2023 Quantifying The Return On Investment For Your Data Team Summary As businesses increasingly invest in technology and talent focused on data engineering and analytics, they want to know whether they are benefiting. So how do you calculate the return on investment for data? In this episode Barr Moses and ...
07.31.2023 Strategies For A Successful Data Platform Migration Summary All software systems are in a constant state of evolution. This makes it impossible to select a truly future-proof technology stack for your data platform, making an eventual migration inevitable. In this episode Gleb Mezhanskiy and Rob Goretsky share ...
07.24.2023 Build Real Time Applications With Operational Simplicity Using Dozer Summary Real-time data processing has steadily been gaining adoption due to advances in the accessibility of the technologies involved. Despite that, it is still a complex set of capabilities. To bring streaming data in reach of application engineers Matteo Pelati ...
07.17.2023 Datapreneurs - How Todays Business Leaders Are Using Data To Define The Future Summary Data has been one of the most substantial drivers of business and economic value for the past few decades. Bob Muglia has had a front-row seat to many of the major shifts driven by technology over his career. In ...
07.09.2023 Reduce Friction In Your Business Analytics Through Entity Centric Data Modeling Summary For business analytics the way that you model the data in your warehouse has a lasting impact on what types of questions can be answered quickly and easily. The major strategies in use today were created decades ago when ...
07.03.2023 How Data Engineering Teams Power Machine Learning With Feature Platforms Summary Feature engineering is a crucial aspect of the machine learning workflow. To make that possible, there are a number of technical and procedural capabilities that must be in place first. In this episode Razi Raziuddin shares how data engineering ...
06.25.2023 Seamless SQL And Python Transformations For Data Engineers And Analysts With SQLMesh Summary Data transformation is a key activity for all of the organizational roles that interact with data. Because of its importance and outsized impact on what is possible for downstream data consumers it is critical that everyone is able to ...
06.18.2023 How Column-Aware Development Tooling Yields Better Data Models Summary Architectural decisions are all based on certain constraints and a desire to optimize for different outcomes. In data systems one of the core architectural exercises is data modeling, which can have significant impacts on what is and is not ...
06.11.2023 Build Better Tests For Your dbt Projects With Datafold And data-diff Summary Data engineering is all about building workflows, pipelines, systems, and interfaces to provide stable and reliable data. Your data can be stable and wrong, but then it isn't reliable. Confidence in your data is achieved through constant validation and ...
06.04.2023 Reduce The Overhead In Your Pipelines With Agile Data Engine's DataOps Service Summary A significant portion of the time spent by data engineering teams is on managing the workflows and operations of their pipelines. DataOps has arisen as a parallel set of practices to that of DevOps teams as a means of ...
05.29.2023 A Roadmap To Bootstrapping The Data Team At Your Startup Summary Building a data team is hard in any circumstance, but at a startup it can be even more challenging. The requirements are fluid, you probably don't have a lot of existing data talent to manage the hiring and onboarding, ...
05.21.2023 Keep Your Data Lake Fresh With Real Time Streams Using Estuary Summary Batch vs. streaming is a long running debate in the world of data integration and transformation. Proponents of the streaming paradigm argue that stream processing engines can easily handle batched workloads, but the reverse isn't true. The batch world ...
05.15.2023 What Happens When The Abstractions Leak On Your Data Summary All of the advancements in our technology is based around the principles of abstraction. These are valuable until they break down, which is an inevitable occurrence. In this episode the host Tobias Macey shares his reflections on recent experiences ...
05.07.2023 Use Consistent And Up To Date Customer Profiles To Power Your Business With Segment Unify Summary Every business has customers, and a critical element of success is understanding who they are and how they are using the companies products or services. The challenge is that most companies have a multitude of systems that contain fragments ...
04.24.2023 Realtime Data Applications Made Easier With Meroxa Summary Real-time capabilities have quickly become an expectation for consumers. The complexity of providing those capabilities is still high, however, making it more difficult for small teams to compete. Meroxa was created to enable teams of all sizes to deliver ...
04.16.2023 Building Self Serve Business Intelligence With AI And Semantic Modeling At Zenlytic Summary Business intellingence has been chasing the promise of self-serve data for decades. As the capabilities of these systems has improved and become more accessible, the target of what self-serve means changes. With the availability of AI powered by large ...
04.10.2023 An Exploration Of The Composable Customer Data Platform Summary The customer data platform is a category of services that was developed early in the evolution of the current era of cloud services for data processing. When it was difficult to wire together the event collection, data modeling, reporting, ...
04.03.2023 Mapping The Data Infrastructure Landscape As A Venture Capitalist Summary The data ecosystem has been building momentum for several years now. As a venture capital investor Matt Turck has been trying to keep track of the main trends and has compiled his findings into the MAD (ML, AI, and ...
03.25.2023 Unlocking The Potential Of Streaming Data Applications Without The Operational Headache At Grainite Summary The promise of streaming data is that it allows you to react to new information as it happens, rather than introducing latency by batching records together. The peril is that building a robust and scalable streaming architecture is always ...
03.19.2023 Aligning Data Security With Business Productivity To Deploy Analytics Safely And At Speed Summary As with all aspects of technology, security is a critical element of data applications, and the different controls can be at cross purposes with productivity. In this episode Yoav Cohen from Satori shares his experiences as a practitioner in ...
03.10.2023 Use Your Data Warehouse To Power Your Product Analytics With NetSpring Summary With the rise of the web and digital business came the need to understand how customers are interacting with the products and services that are being sold. Product analytics has grown into its own category and brought with it ...
03.06.2023 Exploring The Nuances Of Building An Intentional Data Culture Summary The ecosystem for data professionals has matured to the point that there are a large and growing number of distinct roles. With the scope and importance of data steadily increasing it is important for organizations to ensure that everyone ...
02.27.2023 Building A Data Mesh Platform At PayPal Summary There has been a lot of discussion about the practical application of data mesh and how to implement it in an organization. Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building ...
02.19.2023 The View Below The Waterline Of Apache Iceberg And How It Fits In Your Data Lakehouse Summary Cloud data warehouses have unlocked a massive amount of innovation and investment in data applications, but they are still inherently limiting. Because of their complete ownership of your data they constrain the possibilities of what data you can store ...
02.11.2023 Let The Whole Team Participate In Data With The Quilt Versioned Data Hub Summary Data is a team sport, but it's often difficult for everyone on the team to participate. For a long time the mantra of data tools has been "by developers, for developers", which automatically excludes a large portion of the ...
02.06.2023 Reflecting On The Past 6 Years Of Data Engineering Summary This podcast started almost exactly six years ago, and the technology landscape was much different than it is now. In that time there have been a number of generational shifts in how data engineering is done. In this episode ...
01.30.2023 Let Your Business Intelligence Platform Build The Models Automatically With Omni Analytics Summary Business intelligence has gone through many generational shifts, but each generation has largely maintained the same workflow. Data analysts create reports that are used by the business to understand and direct the business, but the process is very labor ...
01.22.2023 Safely Test Your Applications And Analytics With Production Quality Data Using Tonic AI Summary The most interesting and challenging bugs always happen in production, but recreating them is a constant challenge due to differences in the data that you are working with. Building your own scripts to replicate data from production is time ...
01.16.2023 Building Applications With Data As Code On The DataOS Summary The modern data stack has made it more economical to use enterprise grade technologies to power analytics at organizations of every scale. Unfortunately it has also introduced new overhead to manage the full experience as a single workflow. At ...
01.08.2023 Automate Your Pipeline Creation For Streaming Data Transformations With SQLake Summary Managing end-to-end data flows becomes complex and unwieldy as the scale of data and its variety of applications in an organization grows. Part of this complexity is due to the transformation and orchestration of data living in disparate systems. ...
12.29.2022 Increase Your Odds Of Success For Analytics And AI Through More Effective Knowledge Management With AlignAI Summary Making effective use of data requires proper context around the information that is being used. As the size and complexity of your organization increases the difficulty of ensuring that everyone has the necessary knowledge about how to get their ...
12.29.2022 Using Product Driven Development To Improve The Productivity And Effectiveness Of Your Data Teams Summary With all of the messaging about treating data as a product it is becoming difficult to know what that even means. Vishal Singh is the head of products at Starburst which means that he has to spend all of ...
12.26.2022 Simple And Scalable Encryption Of Data In Use For Analytics And Machine Learning With Opaque Systems Summary Encryption and security are critical elements in data analytics and machine learning applications. We have well developed protocols and practices around data that is at rest and in motion, but security around data in use is still severely lacking. ...

Older episodes