Apache Airflow is a system for scheduling and monitoring workflows for data engineering. Airflow can be used to schedule ETL jobs, machine learning work, and script execution. Airflow also gives a developer a high level view into the graph of dependencies for their data pipelines. Chaim Turkel is a backend data architect at Tikal. He

CamelCamelCamel is a tool for tracking prices on items on Amazon.com. The company was launched eleven years ago and is built off of the Amazon Product Advertising API. Daniel Green is one of the founders of CamelCamelCamel, and he joins the show to describe his experience building the product. Amazon and CamelCamelCamel (C3) have a

Google’s codebase is managed in a single monolithic repository. An engineer at Google can explore almost any area of the codebase within the entire company. In order to enable this, Google has built tooling to support the monolithic repo, including a virtual file system and a set of build tools. A monolithic repository is not

Software engineering is harder than it should be. There are many people who have an app idea that they are not sure how to build. Some of these people are highly technical professionals like real estate agents, scientists, and accountants. These professionals learn to use spreadsheets in their day-to-day work. Spreadsheets are also used widely

Relational data systems have evolved from single node instances to complex distributed systems. Almost any database can be accessed through a SQL statement, but the guarantees of these databases can vary in terms of consistency, availability, latency, durability, and financial cost. Relational database systems that explore these different sets of tradeoffs are sometimes categorized as

Relational databases such as Postgres are often used for critical workloads, such as user account data. To run a relational database service in the cloud requires a cloud provider to set up a highly durable, highly available system. Jon Daniel is an infrastructure engineer at Heroku. Jon joins the show to describe the engineering and

Satellite images contain vast quantities of data. By analyzing the contents of satellite images over time, we can identify trends in weather, soil, and agriculture. If we combine that data with ground-level sensors, we can gather a clearer understanding of how chemicals in the air or in the dirt map to how things look from

FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful

RECENT UPDATES: Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more FindCollabs Hackathon has ended–winners will probably be announced by the time this episode airs;

RECENT UPDATES: FindCollabs $5000 Hackathon Ends Saturday April 15th, 2019 New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more Consul is a tool from HashiCorp that allows users to store and retrieve information from a highly available key/value data