
Sign up to save your podcasts
Or


Anthony Diiorio was involved with Ethereum since the earliest days. He was one of the first people to see the Ethereum ideas presented by Vitalik Buterin, and he invested deeply in Ethereum–both financially and by helping to establish the early Ethereum community. Anthony started Decentral in 2014, which is a hub for his projects in the cryptocurrency space, the most impactful project being Jaxx.
Jaxx is a blockchain wallet that can hold multiple different cryptocurrencies. It works by connecting a small client-side application to remote full nodes. The user interface is simple, and Jaxx maintains the full node instances that the small client-side application connects to. We discuss the architecture of Jaxx in more detail during this episode.
We also talk about Anthony’s background–which includes a wide range of businesses: marketing, patio door manufacturing, real estate, and eventually blockchains. Anthony had a wealth of information to provide around entrepreneurship–both inside and outside of the blockchain space.
If you are looking for all 700 episodes of Software Engineering Daily, check out our apps on the iOS or Android app store. We’ve got tons of episodes on blockchains, business, distributed systems, and tons of other topics. If you want to become a paid subscriber to Software Engineering Daily, you can hear all of our episodes without ads–you can subscribe at softwaredaily.com. And all of the code for our apps is open source. If you are looking for an open source community to be a part of, come check out github.com/softwareengineeringdaily.
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
Most applications today run on a cloud provider like AWS. They are built with a framework like Ruby on Rails. They use a set of APIs like Stripe and Twilio for middleware services. This is the era of “web 2.0.”
With decentralized systems, we are starting to get a feel for what “web 3.0” might feel like. The futuristic idea of “web 3.0” works off of the following idea: instead of using a centralized service owned by a single company, you might purchase your computation and storage from a network of nodes. The nodes will be running peer-to-peer software that competes on price.
Fabian Vogelsteller works on Web3.js, a JavaScript library for interfacing with the Ethereum blockchain. He also works on Mist, a browser for Ethereum. Fabian joins the show to discuss the difference between decentralized apps and centralized apps—and to explain why we need to build a bridge between those two worlds.
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
Joseph Bonneau is co-author of Bitcoin and Cryptocurrency Technologies, a popular textbook. At NYU, he works as an assistant professor exploring cryptography and security. His YouTube lessons teaching Bitcoin have hundreds of thousands of views. His material offers clear explanations of how Bitcoin works.
Since Joseph has a clear understanding of the objective facts around Bitcoin, he is the perfect person to ask about the more subjective topics: the common misunderstandings of Bitcoin; the governance tradeoffs between Ethereum and Bitcoin; proof of work vs. proof of stake.
Joseph believes that the early mainstream cryptocurrency solutions will be largely centralized—and that we are likely to move beyond Bitcoin to more efficient currencies. I enjoyed hearing his reasons behind this perspective.
Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA.
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
Bitcoin is an immutable, append-only blockchain ledger that reaches consensus through proof-of-work. The contents of the ledger are financial transactions–people sending and receiving Bitcoin currency to each other.
Since Bitcoin, there have been other cryptocurrencies that have similar properties–like Ethereum and the IPFS/Filecoin system. Similar to Bitcoin, they use a decentralized, proof of work based system with a currency reward system–but the ledger being maintained is not purely financial. A currency is a necessary component to maintaining the blockchain’s validity.
Over the next month, we will be exploring a variety of blockchain-based technologies. Some interviews will be high-level conversations that assume only a familiarity with cryptocurrencies. Some of them will be deeply technical, and assume a strong understanding of Bitcoin and Ethereum. And some episodes, like today’s episode, will be aimed at the developer who is in the process of “going down the rabbit hole.”
If you are finding yourself reading about Bitcoin and Ethereum a few hours every day, but you are still struggling to grasp the basics, this episode is for you. It is meant to be a complement to other introductory resources, such as “Mastering Bitcoin,” by Andreas Antonopoulos.
Cryptocurrency systems are revolutionary–they will unlock completely new applications in the very near future. It’s If you are trying to understand these decentralized currencies and applications, the best place to start is with Bitcoin. That’s why I was happy to have Daniel Van Flymen back on the show.
Daniel was previously on for one of our most popular episodes–”Blockchain Building,” in which he talked about how useful it can be to build a blockchain based system for practice. Today, Daniel discusses the basics of Bitcoin transactions. What happens when you send money? How are transactions represented on the blockchain? How do full nodes and light clients interact with each other?
These are difficult topics to discuss purely over audio, so this episode is best listened to as a companion resource for someone who is studying cryptocurrencies.
If you are looking for an internship, apply to the Software Engineering Daily internship, at softwaredaily.com/jobs. And if you are looking to recruit engineers, you can post jobs for your company there as well–it’s completely free to post jobs and to apply. We are hoping to find interns to contribute to the Software Daily open source project–and if you want to see what we are building, go to SoftwareDaily.com or check out our apps in the iOS or Android app store. They have all 650 of our episodes, with recommendations, related links, discussions and more.
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
Steve Herrod led engineering at VMWare as the company scaled from 30 engineers to 3,000 engineers. After 11 years, he left to become a managing director for General Catalyst, a venture capital firm. Since he has both operating experience and a wide view of the technology landscape as an investor, he is well-equipped to discuss a topic that we have been covering on Software Engineering Daily: the integration of cloud and edge computing.
Today, we think of the cloud as a network of large data centers operated by big players like Google, Amazon, and Microsoft. The cloud is where most of the computation across the world takes place. My smartphone and laptop are “edge” devices. They are lightweight computers that don’t perform much complex processing. I would not be able to run a large production database or a 3 terabyte MapReduce job on my laptop.
The current division of labor makes sense in this world of smart clouds and low-power, low-bandwidth devices. But the devices are getting cheaper, smarter, and more proliferate. Cars, drones, security cameras, sensors and other devices can serve as points of computation that are geographically between the edge devices and the cloud. With more devices between you and the cloud, there is an opportunity to put computation on those devices.
Everyone knows that cloud and edge computing will become intermingled in the coming years. But predicting just how it will play out is nearly impossible. And as an investor, if you bet on something too early, you get the same result as someone who was wrong altogether.
A good analogy for the “cloud and edge” space of investments might be the “smart home.” Everyone knows the smart home is coming eventually, but it’s very hard to tell how long it will be before smart home systems are in widespread use–so it is an open question how to invest in the space.
Summer internship applications to Software Engineering Daily are also being accepted. If you are interested in working with us on the Software Engineering Daily open source project full-time this Summer, send an application to [email protected]. We’d love to hear from you.
If you haven’t seen what we are building, check out softwaredaily.com, or download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily.
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
On Software Engineering Daily, we have been covering the “serverless” movement in detail. For people who don’t use serverless functions, it seems like a niche. Serverless functions are stateless, auto-scaling, event driven blobs of code. You might say “serverless sounds kind of cool, but why don’t I just use a server? It’s a paradigm I’m used to.”
Serverless is exciting not because of what it adds but because of what it subtracts. The potential of serverless technology is to someday not have to worry about scalability at all.
Today, we take for granted that if you start a new company, you are building it on cloud infrastructure. The problem of maintaining server hardware disappeared for 99% of startups, which unlocked a wealth of innovation.
The cloud also simplified scalability for most startups–but there are still plenty of companies that struggle to scale. Significant mental energy is spent on the following questions: How many database replicas do I need? How do I configure my load balancer? How many nodes should I put in my Kafka cluster?
Serverless functions are important because they are an auto-scaling component that sits at a low level. This makes it easy to build auto scaling systems on top of them. Auto scaling databases, queueing systems, machine learning tools, and user applications.
And since the problem is being solved at such a low level, the pricing competitions will also take place at the low level, meaning that systems built on serverless functions will probably see steep declines in costs in the coming years. Serverless compute could eventually become free or nearly free, with the major cloud providers using it as a loss leader to onboard developers to higher level services.
All of this makes for an exciting topic of discussion, that we will be repeatedly covering. Today’s show is with Eduardo Laureano, the principal program manager of Azure Functions. It was a fantastic conversation and we covered applications of serverless, improvements to the “cold start problem,” and how the Azure Functions platform is built and operated. Full disclosure: Microsoft is a sponsor of Software Engineering Daily.
Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA.
Summer internship applications to Software Engineering Daily are also being accepted. If you are interested in working with us on the Software Engineering Daily open source project full-time this Summer, send an application to [email protected]. We’d love to hear from you.
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
When you go to a website where a video is playing, and your video lags, how does the website know that you are having a bad experience?
Problems with video are often not complete failures–maybe part of the video loads, and plays just fine, and then the rest of the video is buffering. You have probably experienced sitting in front of a video, waiting for it to load as the loading wheel mysteriously spins.
Since problems with video are often not complete failures, troubleshooting a problem with a user’s video playback is not as straightforward as just logging whenever a crash occurs. You need to continuously monitor the video playback on every client device and aggregate it in a centralized system for analysis.
The centralized logging system will allow you to separate problems with a specific user from problems with the video service itself. A single user could have bad wifi, or have 50 tabs open with different videos. To identify problems that are caused by the video player rather than the user, you need to capture the playback from every video and every user.
Scott Kidder works at Mux, where he builds a streaming analytics system for video monitoring. In this episode, Scott explains how events make it from a video player onto the backend analytics system running on Kinesis and Apache Flink.
Events from the browser are constantly added to Kinesis (which is much like Kafka). Apache Flink reads those events off of Kinesis and map reduces them to discover anomalies. For example, if 100 users watch a 20 minute cat video, and the video stops playing at minute 12 for all 100 users, there is probably some data corruption in that video. You would only be able to discover that by assessing all users.
Scott and I discussed the streaming infrastructure that he works on at Mux, as well as other streaming systems like Spark, Apache Beam, and Kafka.
This episode is the first in a short series about streaming data infrastructure. I wanted to do some shows in preparation for Strata Data conference in March in San Jose, which I will be attending thanks to a complimentary ticket from O’Reilly. O’Reilly has been kind enough to give me free tickets since Software Engineering Daily started and did not have the money to attend any conferences. If you want to attend Strata, you can use promo code PCSED to get 20% off.
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
At a big enough scale, every software product produces lots of data. Whether you are building an advertising technology company, a social network, or a system for IoT devices, you have thousands of events coming in at a fast pace that you want to aggregate, study and act upon.
For the last decade, engineers have been learning to store and process these vast quantities of data.
The first common technique was to store all your data to HDFS–the Hadoop Distributed File System–and run nightly Hadoop MapReduce jobs across that data. HDFS is cheap (stored on disk), effective (Hadoop had a revolutionary effect on business analysis), and easy to understand (“every night we take all the data from the previous day, analyze it with Hadoop, and send an email report to our analysts”).
The second common technique was the “Lambda Architecture.” The Lambda Architecture used a stream processing system like Apache Storm to process all incoming events as soon as they were created, so that software products could react quickly to the changes occurring in a large scale system. But events would sometimes be processed out of order, or they would get lost due to node failures. To fix those errors, the nightly Hadoop MapReduce jobs would still run, and would reconcile all the problems that might have occurred when the events were processed in the streaming system.
The Lambda Architecture worked pretty well–systems were becoming “real time”, and products like Twitter were starting to feel alive as they were able to rapidly process the massive volume of events on the fly. But managing a system with a Lambda Architecture was painful–you had to manage both a Hadoop cluster and a Storm cluster. You had to make sure that your Hadoop processing did not interfere with your Storm processing.
Today, a newer technique for ingesting and reacting to data has become more common, and is referred to as “streaming analytics.” Streaming analytics is a strategy for performing fast analysis of data coming into a system.
In streaming analytics systems, events are sent to a scalable, durable pubsub system such as Kafka. You can think of Kafka as a huge array of events that have occurred–such as users liking tweets or clicking on ads. Stream processing systems like Apache Flink or Apache Spark read the data from Kafka as if they were reading an array that was being continually appended to.
The sequence of events that get written to Kafka are called “streams”. This can be confusing–with a stream, you imagine this constantly moving, transient sequence of data. That’s partially true, but data will stay in Kafka as long as you want it to. You can set a retention policy for 2 weeks, 2 months, or 2 years. As long as that data is still retained in Kafka, your stream processing system can start reading from any place in the stream.
The stream processing systems like Flink or Spark that read from Kafka are still grabbing batches of data, and processing them in batches. They are reading from the event stream buffer in Kafka, which you can think of as an array. (This is something that confused me for a long time, so if you are still confused, don’t worry, we explain it more in this episode.)
Tugdual Grall is an engineer with MapR. In today’s episode, we explore use cases and architectural patterns for streaming analytics. Full disclosure: MapR is a sponsor of Software Engineering Daily.
In past shows, we have covered data engineering in detail–we’ve looked at Uber’s streaming architecture, talked to Matei Zaharia about the basics of Apache Spark, and explored the history of Hadoop. To find all of our episodes about data engineering , download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily.
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
Pinterest is a visual feed of ideas, products, clothing, and recipes. Millions of users browse Pinterest to find images and text that are tailored to their interests.
Like most companies, Pinterest started with a large monolithic application that served all requests. As Pinterest’s engineering resources expanded, some of the architecture was broken up into microservices and Dockerized, which make the system easier to reason about.
To serve users with better feeds, Pinterest built a machine learning pipeline using Kafka, Spark, and Presto. User events are generated from the frontend, logged onto Kafka, and aggregated to build machine learning models. These models are deployed into Docker containers much like the production microservices.
Kinnary Jangla is a senior software engineer at Pinterest, and she joins the show to talk about her experiences at the company–breaking up the monolith, architecting a machine learning pipeline, and deploying those models into production.
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
Over 12 years of engineering, Box has developed a complex architecture of services. Whenever a user uploads a file to Box, that upload might cause 5 or 6 different services to react to the event. Each of these services is managed by a set of servers, and managing all of these different servers is a challenge.
Sam Ghods is the cofounder and services architect of Box. In 2014, Sam was surveying the landscape of different resource managers, deciding which tool should be the underlying scheduler for deploying services at Box. He chose Kubernetes because it was based on Google’s internal Borg scheduling system.
For years, engineering teams at companies like Facebook and Twitter had built internal scheduling systems modeled after Borg. When Kubernetes arrived, it provided an out-of-the-box tool for managing infrastructure like Google would.
In today’s episode, Sam describes how Box began its migration to Kubernetes, and what the company has learned along the way. It’s a great case study for people who are looking at migrating their own systems to Kubernetes.
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
From the publisher's feed