
Sign up to save your podcasts
Or


Java has been popular since the 90s, when it started to be used as a programming language for enterprises.
Today, Java is still widely deployed, but the infrastructure environment is dramatically different. Java is often deployed to containers in the cloud. If those containers can share resources, then those containers can share the same underlying Java infrastructure.
Java 13 is the most recent public release of Java. The new features in Java 13 reflect the changing demands of modern application developers. Georges Saab is an engineer with Oracle who has been working on Java for more than a decade. He joins the show to discuss how Java development patterns are changing, and how the language is evolving to accommodate those changes, including discussion of garbage collection and dynamic application class data sharing.
Relational databases provide durable transactional systems for storing data. The relational model has existed for decades, but the requirements for a relational database have changed.
Modern applications have requirements for high volumes of data that do not fit onto a single machine. When a database gets too big to fit on a single machine, that database needs to be sharded into smaller subsets of the data. These database shards are spread across multiple machines, and as the database grows, the database can be resharded to scale to even more machines.
To ensure durability, a database needs to be replicated. The database needs to be able to survive any single machine losing power or getting destroyed.
Sharding and replication allow a relational database to be scalable, durable, and highly available. There are many ways to build sharding and replication into a database. Karthik Ranganathan and Sidharth Choudhury are engineers with YugabyteDB, a distributed SQL database.
In today’s episode, we discuss the modern requirements of a distributed SQL database, and compare the applications of distributed SQL to those of other systems such as Cassandra and Hadoop. We also talk through the competitive market of cloud-based distributed SQL providers such as Google Cloud Spanner and Amazon Aurora. YugabyteDB is an open source database that competes with these other relational databases. Full disclosure: YugabyteDB is a sponsor of Software Engineering Daily.
Originally published July 14, 2016
EconTalk is a weekly economics podcast that has been going for a decade. On EconTalk, Russ Roberts brings on writers, intellectuals, and entrepreneurs for engaging conversations about the world as seen through the lens of economics.
Russ Roberts is today’s guest, and it is a treat because I have been listening to EconTalk since 2006 and it was a central point of inspiration for what Software Engineering Daily has become. On this episode, we talk about how software impacts the world economically, from bitcoin’s promise of zero cost transactions to the opportunities and regulatory challenges of the software-enabled gig economy.
Originally published May 24, 2018
When a user takes a ride on Uber, the app on the user’s phone is communicating with Uber’s backend infrastructure, which is writing to a database that maintains the state of that user’s activity. This database is known as a transactional database or “OLTP” (online transaction processing). Every active user and driver and UberEATS restaurant is writing data to the transactional data store.
Periodically, that data is copied from the transactional data system to a different data storage system, where that data can be queried for large-scale data analysis. For example, if a data scientist at Uber wants to get the average amount of miles that a given user rode in February, that data scientist would issue a query to the analytical data cluster.
Uber uses the Hadoop distributed file system (HDFS) to store analytical data. On this file system, Uber has a version history of all of the company’s useful historical data. Trip history, rider activity, driver activity–every data point that is in the transactional database–but in a file format that is easier to query for large scale processing. This file format is known as Parquet.
Data scientists, machine learning engineers, and real-time application developers all depend on the massive quantities of data that are stored in these Parquet files on Uber’s HDFS cluster. To simplify the access of that data by many different clients, Uber uses Presto, an analytical query engine originally built at Facebook.
Presto translates SQL queries into whatever query language is necessary to access the underlying storage medium–whether that storage system is an ElasticSearch cluster, a set of Parquet files, or a relational database. Presto is useful because it simplifies the relationship between data engineers and the application developers who are building on top of the data engineering infrastructure.
In today’s show, Zhenxiao Luo joins to give an end-to-end description of Uber’s data infrastructure–from the ingest point of the OLTP database to the OLAP data storage system on HDFS, to the wide range of data systems and applications that run on top of that OLAP data.
Originally published May 24, 2018
When a user takes a ride on Uber, the app on the user’s phone is communicating with Uber’s backend infrastructure, which is writing to a database that maintains the state of that user’s activity. This database is known as a transactional database or “OLTP” (online transaction processing). Every active user and driver and UberEATS restaurant is writing data to the transactional data store.
Periodically, that data is copied from the transactional data system to a different data storage system, where that data can be queried for large-scale data analysis. For example, if a data scientist at Uber wants to get the average amount of miles that a given user rode in February, that data scientist would issue a query to the analytical data cluster.
Uber uses the Hadoop distributed file system (HDFS) to store analytical data. On this file system, Uber has a version history of all of the company’s useful historical data. Trip history, rider activity, driver activity–every data point that is in the transactional database–but in a file format that is easier to query for large scale processing. This file format is known as Parquet.
Data scientists, machine learning engineers, and real-time application developers all depend on the massive quantities of data that are stored in these Parquet files on Uber’s HDFS cluster. To simplify the access of that data by many different clients, Uber uses Presto, an analytical query engine originally built at Facebook.
Presto translates SQL queries into whatever query language is necessary to access the underlying storage medium–whether that storage system is an ElasticSearch cluster, a set of Parquet files, or a relational database. Presto is useful because it simplifies the relationship between data engineers and the application developers who are building on top of the data engineering infrastructure.
In today’s show, Zhenxiao Luo joins to give an end-to-end description of Uber’s data infrastructure–from the ingest point of the OLTP database to the OLAP data storage system on HDFS, to the wide range of data systems and applications that run on top of that OLAP data.
Originally published January 4, 2019
When a user makes a request to product like The New York Times, that request hits an API gateway. An API gateway is the entry point for an external request. An API gateway serves several purposes: authentication, security, routing, load balancing, and logging.
API gateways have grown in popularity as applications have become more distributed, and companies offer a wider variety of services. If an API is public, and anyone can access it, you might need to apply rate limiting so that users cannot spam the API. If the API is private, the user needs to be authenticated before the request is fulfilled.
Kong is a company that builds infrastructure for API management. The Kong API gateway is a widely used open source project, and Kong is a company built around supporting and building on top of the API gateway.
Marco Palladino is the co-founder and CTO of Kong. He joins the show to tell the story of starting Kong eight years ago, and how the API gateway product evolved out of an API marketplace. Marco also discusses the architecture of Kong and his vision for how the product will develop in the future–including the Kong service mesh.
Originally published June 7, 2018
Moore’s Law states that the number of transistors in a dense integrated circuit doubles about every two years. Moore’s Law is less like a “law” and more like an observation or a prediction.
Moore’s Law is ending. We can no longer fit an increasing amount of transistors in the same amount of space with a highly predictable rate. Dennard scaling is also coming to an end. Dennard scaling is the observation that as transistors get smaller, the power density stays constant.
These changes in hardware trends have downstream effects for software engineers. Most importantly–power consumption becomes much more important.
As a software engineer, how does power consumption affect you? It means that inefficient software will either run more slowly or cost more money relative to our expectations in the past. Whereas software engineers writing code 15 years ago could comfortably project that their code would get significantly cheaper to run over time due to hardware advances, the story is more complicated today.
Why is Moore’s Law ending? And what kinds of predictable advances in technology can we still expect?
John Hennessy is the chairman of Alphabet. In 2017, he won a Turing award (along with David Patterson) for his work on the RISC (Reduced Instruction Set Compiler) architecture. From 2000 to 2016, he was the president of Stanford University.
John joins the show to explore the future of computing. While we may not have the predictable benefits of Moore’s Law and Dennard scaling, we now have machine learning. It is hard to plot the advances of machine learning on any one chart (as we explored in a recent episode with OpenAI). But we can say empirically that machine learning is working quite well in production.
If machine learning offers us such strong advances in computing, how can we change our hardware design process to make machine learning more efficient?
As machine learning training workloads eat up more resources in a data center, engineers are developing domain specific chips which are optimized for those machine learning workloads. The Tensor Processing Unit (TPU) from Google is one such example. John mentioned that chips could become even more specialized within the domain of machine learning. You could imagine a chip that is specifically designed for a LSTM machine learning model.
There are other domains where we could see specialized chips–drones, self-driving cars, wearable computers. In this episode, John describes his perspective on the future of computing, and offers some framework for how engineers can adapt to that future.
Originally published November 4, 2016
Indie Hackers is a website that profiles independent developers who have made profitable software projects, usually without raising any money. These projects make anywhere from a few hundred dollars a month to more than $100,000 as in the case with park.io, one of the services profiled by Indie Hackers.
Courtland Allen is the creator, engineer, and interviewer behind Indie Hackers. For each business that is profiled by Indie Hackers, Courtland conducts a short interview with the founder.
Courtland joins the show to discuss the changing trends that are making it easier to bootstrap a software business if you are a capable developer–or even if you are a nontechnical person who understands how software works. Since Courtland and I are both in the business of interviewing engineers, we had a lot to talk about, and this is a fantastic episode.
Originally published November 4, 2016
Indie Hackers is a website that profiles independent developers who have made profitable software projects, usually without raising any money. These projects make anywhere from a few hundred dollars a month to more than $100,000 as in the case with park.io, one of the services profiled by Indie Hackers.
Courtland Allen is the creator, engineer, and interviewer behind Indie Hackers. For each business that is profiled by Indie Hackers, Courtland conducts a short interview with the founder.
Courtland joins the show to discuss the changing trends that are making it easier to bootstrap a software business if you are a capable developer–or even if you are a nontechnical person who understands how software works. Since Courtland and I are both in the business of interviewing engineers, we had a lot to talk about, and this is a fantastic episode.
New software abstractions always take advantage of the abstractions that have been built before.
Software libraries allow us to import code that sits on the same host as a new program. Open source software let us copy and paste existing code, or clone entire repositories. Cloud providers offer hosted tools and APIs that we can leverage to develop scalable, easy-to-use infrastructure.
When existing pieces of software are built into new software, the existing software becomes a dependency. Managing those dependencies is an engineering problem. Mya Pitzeruse is the founder of deps.cloud, a project with the goal of improving dependency changes across a company’s ecosystem. In today’s show, we talk about the modern dependency issues of a large company, and her perspective on how to address them. Mya has developed the project in public on FindCollabs, and we also spend some time talking about building in the open.
From the publisher's feed