Software Engineering Daily

Self-driving, electric cars will someday outnumber traditional automobiles on the road. As transportation becomes autonomous, it is hard to imagine an industry that will not be affected by the downstream effects of this change.

These cars will likely be managed by fleet operators like Lyft and Uber. We will need fewer cars, and the amount of space dedicated to those cars will shrink dramatically. Parking lots, massive roads, and gas stations will be reclaimed or repurposed. City planning departments will have to devise entirely new strategies.

As the self-driving cars reach consumer availability, an intricate supply chain for these cars will develop. When smartphones became mass-produced, the costs of GPS devices, accelerometers, and other small components dropped steeply. A consequence of the smartphone supply chain was that other devices like consumer drones became affordable. The self-driving car supply chain will lead to the mass production of building blocks for other new devices.

With fewer automotive fatalities, the economics of the car insurance industry might collapse completely. At a minimum, the costs of car insurance will likely shift to the fleet operators, who can purchase that car insurance at prices factoring in their large risk pool.

Frank Chen is a deal and research partner with Andreessen Horowitz. In a series of presentations on the Autonomy Ecosystem, Frank explores the effects of our impending shift to self-driving electric cars. His analysis considers changes to energy infrastructure, the competitive landscape of software companies, and a range of other topics. Frank joins the show to discuss autonomous vehicles and the side effects of widespread autonomous deployments.

The post Autonomy with Frank Chen appeared first on Software Engineering Daily.

When a user takes a ride on Uber, the app on the user’s phone is communicating with Uber’s backend infrastructure, which is writing to a database that maintains the state of that user’s activity. This database is known as a transactional database or “OLTP” (online transaction processing). Every active user and driver and UberEATS restaurant is writing data to the transactional data store.

Periodically, that data is copied from the transactional data system to a different data storage system, where that data can be queried for large-scale data analysis. For example, if a data scientist at Uber wants to get the average amount of miles that a given user rode in February, that data scientist would issue a query to the analytical data cluster.

Uber uses the Hadoop distributed file system (HDFS) to store analytical data. On this file system, Uber has a version history of all of the company’s useful historical data. Trip history, rider activity, driver activity–every data point that is in the transactional database–but in a file format that is easier to query for large scale processing. This file format is known as Parquet.

Data scientists, machine learning engineers, and real-time application developers all depend on the massive quantities of data that are stored in these Parquet files on Uber’s HDFS cluster. To simplify the access of that data by many different clients, Uber uses Presto, an analytical query engine originally built at Facebook.

Presto translates SQL queries into whatever query language is necessary to access the underlying storage medium–whether that storage system is an ElasticSearch cluster, a set of Parquet files, or a relational database. Presto is useful because it simplifies the relationship between data engineers and the application developers who are building on top of the data engineering infrastructure.

In today’s show, Zhenxiao Luo joins to give an end-to-end description of Uber’s data infrastructure–from the ingest point of the OLTP database to the OLAP data storage system on HDFS, to the wide range of data systems and applications that run on top of that OLAP data.

The post Uber’s Data Platform with Zhenxiao Luo appeared first on Software Engineering Daily.

A sample of the human voice is a rich piece of unstructured data. Voice recordings can be turned into visualizations called spectrograms. Machine learning models can be trained to identify features of these spectrograms. Using this kind of analytic strategy, breakthroughs in voice analysis are happening at an amazing pace.

Rita Singh researches voice at Carnegie Mellon University. Her work studies the high volume of latent data that is available in the human voice. As she explains, just a small fragment of a human voice can be used to identify who a speaker is. Your voice is as distinctive as your fingerprint.

Your voice can also reveal medical conditions. Features of the human voice can be strongly correlated with psychiatric symptom severity, and potentially heart disease, cancer, and other illnesses. The human voice can even suggest a person’s physique–your height, weight, and facial features.

In this episode, Rita explains the machine learning techniques that she uses to uncover the hidden richness of the human voice.

The post Voice with Rita Singh appeared first on Software Engineering Daily.

Mesos is a system for managing distributed systems. The goal of Mesos is to help engineers orchestrate resources among multi-node applications like Spark. Mesos can also manage lower level schedulers like Kubernetes. A common misconception is that Mesos aims to solve the same problem as Kubernetes, but Mesos is a higher level abstraction.

Ben Hindman co-founded Mesosphere to bring the Mesos project to market. Large enterprises like Uber, Netflix, and Yelp use Mesosphere for resource management. Before he started the company, Ben worked in the Berkeley AMP Lab, a research program where the Spark and Tachyon projects were also born.

At this point, he has spent significant time in both academia and industry. This conversation spans distributed systems theory, history, and practice. Ben and I spoke at KubeCon 2018 in Copenhagen–which was an amazing conference. We were both amazed at how big the audience for Kubernetes has gotten, and the pace at which the technology is advancing.

Today, Kubernetes is mostly used for scheduling containerized applications that engineers have built themselves. But there will be higher level tools that use Kubernetes as a building block. Much like Zookeeper was used as a building block for Hadoop, Kubernetes will be used to build serverless applications and distributed databases.

Once you are using a distributed database built on Kubernetes, you don’t want to think about the container orchestration–you want to think about the raw storage and CPU requirements for that database. This is one reason why Mesos is so compelling. Since Kubernetes creates an increased cardinality of distributed systems, it’s good to know that there is a framework built to manage those higher level applications.

The post Cluster Schedulers with Ben Hindman appeared first on Software Engineering Daily.

Technology is pushing us rapidly toward a future that is impossible to forecast. We try to imagine what that future might look like, and we can’t help having our predictions shaped by the media we have consumed.

1984, Terminator, Gattaca, Ex Machina, Black Mirror–all of these stories present a dystopian future. But if you look around the world, the most successful technologists are mostly guided by a sense of optimism. Technologists themselves are mostly idealistic–they see the future through a utopian lens. Popular media largely tells a different story: that we are headed for a dystopian world.

Why is there such a gulf in the level of idealism between technologists and the media?

Mike Solana found himself asking that question on a regular basis during his work at Founder’s Fund, where he is a vice president. Founder’s Fund has a bias toward funding difficult, cutting-edge technology like gene editing, robotics, and nuclear energy. This technology that Mike was seeing made him excited about the future–which led to his creation of the podcast “Anatomy of Next.”

“Anatomy of Next” has explored biology, robotics, nuclear energy, superintelligence, and the nature of reality. Soon the podcast will be exploring how our civilization will explore and settle the solar system–specifically Mars.

I’ve listened through the entire first season of the show twice and enjoyed it so much because Mike explores questions that are on the border of philosophy and technology–questions about the nature of reality, and what makes us human–and nobody can give perfect answers to these questions. But Mike interviews top experts on the show, which provides us with a framework. Guests on “Anatomy of Next” include Nick Bostrom (the author of Superintelligence), George Church (a pioneer in gene editing), and Palmer Luckey (the founder of VR company Oculus).

Mike joins the show to talk about why he started “Anatomy of Next,” and his own perspective on the future.

The post Technology Utopia with Michael Solana appeared first on Software Engineering Daily.

Google’s central system for managing to compute resources is called Borg. On Borg, millions of Linux containers process a wide variety of workloads. When a new application is spun up, Borg provides that application with the resources it needs.

Workloads at Google usually fall into one of two distinct categories: long-running application workloads (such as Gmail) and batch workloads (such as a MapReduce job). In the early days of Google, the long-lived workloads were scheduled onto a system called “BabySitter” and the batch workloads were scheduled onto a system called “Global Work Queue.”

Borg was the first cluster manager at Google designed to service both long-running and batch workloads from a single system. The second cluster manager at Google was Omega, a project that was created to improve the engineering behind Borg. The innovations of Omega improved the efficiency and architecture of Borg.

More recently, Kubernetes was created as an open source implementation of the ideas pioneered in Borg and Omega. Google has also built a Kubernetes as a service offering that companies use to run their infrastructure in the same way that Google does.

Brian Grant is an engineer at Google who has seen the iteration of all three cluster management systems that have come out of Google. He joins the show to discuss how the workloads at Google have changed over time, and how his perspective on how to build and architect distributed systems has evolved. Full disclosure: Google is a sponsor of Software Engineering Daily.

The post Google Cluster Evolution with Brian Grant appeared first on Software Engineering Daily.

Machine learning tools are rapidly maturing. TensorFlow gave developers an open source version of Google’s internal machine learning framework. Cloud computing provides a cost effective, accessible way of training models. Edge computing allows for low latency deployments of models.

But even if you are a kid with a laptop who has learned all the machine learning algorithms, read all of the deep learning textbooks, and figured out how to use AWS, all of the tooling and education in the world doesn’t change the fact that you still need data to build models.

This illustrates why we need data-as-a-service.

A kid with a laptop has access to infrastructure-as-a-service, platform-as-a-service, and software-as-a-service. As these tools build on each other, there has been an explosion of high-leverage software products. But the world of data sets remains crude and underdeveloped.

Think about some data sets you could take advantage of the number of emergency room patients that come into a hospital with chest pain; the size of the average coffee mug; the principal component breakdown of sidewalk concrete in San Francisco.

SafeGraph is a company that offers data sets as a service. Auren Hoffman is the CEO of SafeGraph, and he joins the show to discuss why he started building SafeGraph and how he thinks about the state of publicly accessible data.

Auren was previously on the podcast, and I always enjoy talking to him–this was a great episode and I think you will like it as well. Full disclosure: LiveRamp is a sponsor of Software Engineering Daily, LiveRamp being the company that Auren created prior to SafeGraph.

Raj Chetty economic papers

Paul Graham “Keep Your Identity Small”

Auren Hoffman on Quora

The post SafeGraph with Auren Hoffman appeared first on Software Engineering Daily.

“The Federal Reserve System is fraudulent. Whatever its stated purpose, its effective purpose is to create a mechanism of deficit spending by politicians, through the insidious invisible taxation of monetary debasement (aka inflation).”

These are the words of Erik Voorhees, the CEO of crypto financial exchange ShapeShift. Long before he started ShapeShift, Erik was opposed to some of the core principles of the global financial system, in which he sees the US dollar as a means of control. As an early adopter of Bitcoin, he saw a way to make financial transactions without using fiat currency.

Erik’s company ShapeShift allows users to convert different digital currencies between each other. Because it only makes exchanges of currencies and does not hold much currency at any time, ShapeShift is resilient to hacking.

In this episode, Erik and I discussed his economic philosophy, and how that informs his affinity for cryptocurrencies. Erik also describes the architecture of ShapeShift and gives some advice on how to think about building businesses around cryptocurrencies. ShapeShift has had a few near-death experiences, like any startup, and there is a useful story in this episode about how to survive and recover from a serious business setback.

The post ShapeShift with Erik Voorhees appeared first on Software Engineering Daily.

Cryptocurrency speculation has pulled in a large population of people who do not know what they are investing in. If you hear about an investment of $1000 turning into $1M, it’s tempting to get sucked in yourself.

For most of these everyday people, the game is completely rigged. A large percentage of market activity is driven by “pump and dumps.” A pump and dump is a conspiracy to trick investors into buying a currency.

An insider group commits the pump and dump. This is accomplished by purchasing the currency ahead of time, then promoting it via Twitter, Telegram, and Reddit. The outsiders fall victim to the promotion of the currency and buy it after the fast run-up in value. The currency then crashes, and the outsiders are left “holding the bag.”

Pump and dumps are not a new phenomenon—they have happened with worthless penny stocks. One thing that is new is the ease with which new cryptocurrencies are being created. Launching an ICO is easy. Marketing it is cheap. Pumping and dumping has never been more accessible. And buying them is quite easy as well. This has led to a perfect storm of naive investment capital.

Bruno Skvorc is the CEO and owner of Bitfalls, a site with blog posts, news, and information about cryptocurrencies. He wrote a post called “The Anatomy of a Pump and Dump Group,” which details how cryptocurrency pump and dumps have been used to swindle investors out of millions of dollars.

Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA.

If you are looking for an internship, apply to the Software Engineering Daily internship, at softwaredaily.com/jobs. And if you are looking to recruit engineers, you can post jobs for your company there as well–it’s completely free to post jobs and to apply. We are hoping to find interns to contribute to the Software Daily open source project–and if you want to see what we are building, go to SoftwareDaily.com or check out our apps in the iOS or Android app store. They have all 650 of our episodes, with recommendations, related links, discussions, and more.

The post Crypto Pump and Dumps with Bruno Skvorc appeared first on Software Engineering Daily.

Joseph Bonneau is co-author of Bitcoin and Cryptocurrency Technologies, a popular textbook. At NYU, he works as an assistant professor exploring cryptography and security. His YouTube lessons teaching Bitcoin have hundreds of thousands of views. His material offers clear explanations of how Bitcoin works.

Since Joseph has a clear understanding of the objective facts around Bitcoin, he is the perfect person to ask about the more subjective topics: the common misunderstandings of Bitcoin; the governance tradeoffs between Ethereum and Bitcoin; proof of work vs. proof of stake.

Joseph believes that the early mainstream cryptocurrency solutions will be largely centralized—and that we are likely to move beyond Bitcoin to more efficient currencies. I enjoyed hearing his reasons behind this perspective.

Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA.

The post Bitcoin’s Future with Joseph Bonneau appeared first on Software Engineering Daily.