
Sign up to save your podcasts
Or


When engineers are writing code, they are manipulating objects. You might have a user object represented on your computer, and that user object has several different fields—a name, a gender, and an age.
When you want to send that object across the network to a different computer, the object needs to be turned into a sequence of 1s and 0s that will travel efficiently across the network. This is known as “serialization.”
As the user object sits on your computer, it is represented in 1s and 0s. You could just send that same representation over the wire. But we use efficient serialization to send it over the network in a more compact format. We also have to make sure that when we send that object to another service, the other service knows how to deserialize it, and turn it back into a format that we can operate on at the application level.
Protocol buffers are a serialization protocol that originated at Google. Protocol buffers created a standardized interface for efficiently passing data between services. When Kenton Varda worked at Google, he was the tech lead for protocol buffers, and he joins the show to explain how protobufs work—and a newer serialization protocol that Kenton led: Cap’n Proto.
You can expect to walk away from this episode with an understanding of how serialization protocols work, and the design tradeoffs you can make when creating a serialization protocol.
We also touched on a startup that Kenton founded, called Sandstorm, and how he eventually found himself at Cloudflare, where he works on Cloudflare workers. With these topics, we did not go as deep as I would have liked, and I look forward to having Kenton back on in the near future.
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
Duolingo is a language learning platform with over 200 million users. On a daily basis millions of users receive customized language lessons targeted specifically to them. These lessons are generated by a system called the session generator.
Andre Kenji Horie is senior engineer at Duolingo. He wrote about the process of rewriting the session generator, moving from Python to Scala and changing architecture at the same time. In this episode Adam Bell talks with him about the reasons for the rewrite, what drove them to move to Scala and the experience of moving from one technology stack to another.
Rewriting Doulingo’s Engine in Scala
Jobs at Duolingo
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
The values system of a company guides the actions of the engineers who work at that company.
Some companies value open communication and a flat organization where anybody can talk to anyone else. Other companies encourage hierarchy and secrecy, so that employees are focused on their specific section of the company.
Some companies take themselves seriously, and have a work environment that is as stoic as the military. Other companies pride themselves on having good beer and a friendly, laid back atmosphere.
When company values are properly defined, the values can be used as reference points when making decisions. At Amazon, one of the core company values is “bias for action.” As an engineer, you are often in a situation where you can wait for more information, or you can start a project with an incomplete picture for how you will finish it. The “bias for action” lets you know that you should usually start the project despite having an incomplete picture.
Another use of a company values system is for hiring.
When a company publishes their values, prospective employees can use those stated values as a way to know if they would be a good cultural fit. For example “move fast and break things” was a value that allowed Facebook to ship new products faster than any other company before it. But the speed of movement is not for everyone. Some engineers like to have their code unit tested, and free of all bugs before shipping to production.
Every company has values that define their company. And every engineer has values that define how they want to work.
Lynne Tye started her company Key Values as a platform to index companies by their values systems. This allows engineers to find companies that are a good cultural fit for their values system. Lynne joins the show today to explain how engineers and companies define their values systems, and how that affects the outcomes of engineering organizations.
Lynne also talks about her time at HomeJoy, one of the first companies in the “gig economy”. HomeJoy was an on-demand house cleaning service that grew extremely fast, but ultimately went under due to lawsuits. The challenges of HomeJoy were a predictor of the challenges later faced by Uber and Airbnb, and it was fascinating to hear Lynne reflect on her time spent managing operations at HomeJoy–which was about as operationally intensive a company as you can imagine!
Thanks to Courtland Allen for the intro to Lynne, and if you haven’t checked out the Indie Hackers podcast, which is hosted by Courtland, you should subscribe to it. Indie Hackers breaks down the engineering and business models behind small software companies–it’s one of my favorite shows.
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
Ten years ago, if you wanted to build software, you probably needed to know how to write code. Today, the line between “technical” and “non-technical” people is blurring.
Website designers can make a living building sites for people on WordPress or Squarespace–without knowing how to write code. Salesforce integration experts can help a sales team set up complicated software–without knowing how to write code. Shopify experts can set up an ecommerce store to your exact specifications–without knowing how to write code.
WordPress, Squarespace, Salesforce, and Shopify are all fantastic services–but they are not compatible with each other. I can’t install a WordPress plugin on Salesforce.
Now imagine this from the point of view of plugin creators. Plugin creators make easy ways to integrate different pieces of software together. Take PayPal as an example. PayPal wants to make it easy for software builders to integrate with their API.
One plugin that PayPal has is a button that says “Pay with PayPal.” If I am a developer at PayPal, and I am building a button that people should be able to easily put on their webpage so that their users can pay with PayPal, I have to create a button that is compatible with WordPress, and Squarespace, and Wix, and Weebly, and GoDaddy, and Blogger, and all the other website builders that I might want to integrate with.
In 2014, Zack Bloom started a company called Eager. Eager was a cloud app marketplace which allowed app developers to make flexible plugins that non-technical users could drag and drop into their site without technical expertise.
In order for these non-technical users to add any apps from the Eager marketplace to their webpage, they had to drop in a line of JavaScript–which is, unfortunately, a significant hurdle for a nontechnical user.
Eager proved to be a useful distribution mechanism for plugin developers who could write a plugin once and get distributed to multiple plugin marketplaces. But Eager was not as widely used as a way to directly drag and drop plugins onto sites.
The question was: how do you build a marketplace for non-technical users to add plugins to any website without forcing the non-technical user to write code? How do you make editing any website as easy as a WYSIWYG editor?
The CDN turns out to be the perfect distribution platform for these kinds of apps. Users already integrate with a CDN, so the CDN can do the work of inserting the code that allows the plugins to be added to a user’s webpage.
Because of the opportunity for the integration between a plugin marketplace and a CDN, Eager was acquired by Cloudflare, and Eager became Cloudflare apps. Zack Bloom joins the show today to discuss the motivations for his company, the engineering behind building a cloud app marketplace, and the acquisition process of his company Eager.
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
Remember when the best game you could play on your phone was Snake?
In 1998, Snake was preloaded on Nokia phones, and it was massively popular. That same year Half-Life won game of the year on PC. Metal Gear Solid came out for Playstation. The first version of Starcraft also came out in 1998.
In 1998, few people would have anticipated that games with as much interactivity as Starcraft would be played on mobile phones twenty years later. Today, mobile phones have the graphics and processing power of a desktop gaming PC from two decades ago.
But one thing still separates desktop gaming from mobile gaming: the network.
With desktop gaming, users have a reliable wired connection that keeps their packets moving over the network with speeds that let them compete with other users. With mobile gaming, the network can be flaky. How do we architect real-time strategy games that can be played over an intermittent network connection?
Yan Cui is an engineer at Space Ape Games, a company that makes interactive multiplayer games for mobile devices. In a previous episode, Yan described his work re-architecting a social networking startup where the costs had gotten out of control. Yan has a skill for describing software architecture and explaining the tradeoffs.
When architecting a multiplayer mobile game, there are many tradeoffs to consider. What do you build and what do you buy? Do you centralize your geographical deployment to make it easier to reconcile conflicts, or do you spread your server deployment out globally? What is the interaction between the mobile clients and the server?
The question of interaction between client and server for a mobile game has lessons that are important for anyone building a highly interactive mobile application.
For example, think about Uber. When I make a request for a car, I can look at my phone and see the car on the map, slowly approaching me. The driver can look at his phone and see if I move across the street.
This is accomplished by synchronizing the data from the driver’s phone and my phone in a centralized server, and sending the synchronized state of the world out to me and the driver. How much data does the centralized server need to get from the mobile phones? How often does it need to make those requests?
The answers to these questions will vary based on bandwidth, device type, phone battery life, and other factors.
There are similar problems in mobile game engineering, when users are in different players on a virtual map. They are fighting each other, trying to avoid enemies, trying to steal power ups from each other. Mobile games can be even more interactive than a ridesharing app like Uber, so the questions of data synchronization can be even harder to answer.
On Software Engineering Daily, we have explored the topic of real-time synchronization in our past shows about the infrastructure of Uber and Lyft. To find these old episodes, you can download the Software Engineering Daily app for iOS and for Android. In other podcast players, you can only access the most recent 100 episodes. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help.
Yan Cui’s new video course: AWS Lambda in Motion
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
Developers can build networked applications today without having to deploy their code to a server. These “serverless” applications are constructed from managed services and functions-as-a-service.
Managed services are cloud offerings like database-as-a-service, queueing-as-a-service, or search-as-a-service. These managed services are easy to use. They take care of operational burdens like scalability and outages. But managed services typically solve a narrow use case. You can’t build an application entirely out of managed services.
Managed services are scalable and narrow. Functions-as-a-service are scalable and flexible.
With managed services, you make remote calls to a service with a well-defined API. With functions-as-a-service, you can deploy your own code. But functions-as-a-service execute against transient, unreliable compute resources. They aren’t a good fit for low latency computation, and the code you run on them should be stateless.
Managed services and functions-as-a-service are the perfect complements.
Managed services provide you with well-defined server abstractions that every application needs—like databases, search indexes, and queues. Functions as a service offer flexible “glue code” that you can use to create custom interactions between the managed services.
The term “serverless” is used to describe the applications that are built entirely with managed services and functions as a service.
Serverless applications are dramatically simpler to build and easier to operate than cloud applications of the past. The costs of managed services can get expensive, but the costs of functions as a service can cost 1/10th of what it might take to run a server that is handling your requests.
Whether the size of your bill will increase or decrease as your company becomes “serverless” is less of an issue than the fact that your employees will be more productive: serverless applications have less operational burden, so developers spend more time architecting and implementing software.
It has been 5 years since the Netflix infrastructure team was talking about the aspirational goal of a “no-ops” software culture. Your software should be so well-defined that you do not need regular intervention of ops staff to reboot your servers and reconfigure your load balancers. Serverless is a newer way of moving operational expense into capital expense.
Today’s guest Randall Hunt is a senior technical evangelist with Amazon Web Services. He travels around the world meeting developers and speaking at conferences about AWS Lambda, the functions as a service platform from Amazon. Randall has given some excellent talks about how to architect and build serverless applications (which I will add to the show notes), and today we explore those application patterns further.
Serverless Services – Randall Hunt
Randall Hunt at AWS Summit Seoul
Serverless, What is it Good For? Randall Hunt
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
A company’s approach to data can make or break the business.
In the past, data was static. There was not much data, it sat in Excel, and it was interacted with on a nightly or monthly basis. Now, data is dynamic, real time and huge. To tap into available data, many industries have oriented themselves to becoming data intensive. With many new industry sectors becoming data driven, a new field called data science emerged.
As a new field, data science has attracted a lot of attention from professionals with diverse backgrounds. Describing what is data science and who is a data scientist is not easy. As technologies surrounding the field continue to evolve and new verticals are added, the discourse surrounding the field has attracted different voices putting forward their definition of the field.
In this episode, Zacharias Voulgaris joins guest host Sid Ramesh to discuss the developments in the field. He is the author of several data science books, and in today’s conversation Zacharias explains what he means by the data science mindset–including trends and misconceptions that people have on the field.
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
When I log into my bank account from my laptop, I first enter my banking password. Then the bank sends a text message to my phone with a unique code, and I enter that code into my computer to finish the login. This login process is two-factor authentication. I am proving my identity by entering my banking password (the first factor) and validating that I am in control of my phone (the second factor) by receiving that text message.
But in order to log in from my laptop, I need to be in control of my laptop. The laptop itself is a factor. With the laptop and my password, I have two factors. I might not actually need the phone as a factor.
Praneet Sharma is the CEO of Keyless, a product that moves 2-factor authentication into the browser. Praneet joins the show to discuss how all kinds of authentication work: multi-factor authentication, single sign on, and Yubikey. We use this discussion of authentication methods to help explain why it actually could make sense for some people to be doing 2-factor authentication without requiring people to take out their phone.
We also explore recent security breaches like Target, Equifax and Yahoo–and the industry of security software sold to developers. I see giant banners for security software companies every time I go into the San Francisco airport, and Praneet explained to me some of the products that these kinds of companies are selling.
Praneet has joined the show in a previous episode to talk about advertising fraud. He also works with Shailin Dhar at Method Media Intelligence.
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
Most user interfaces that we interact with are not animated. We click on a button, and a form blinks into view. We click a link and the page abruptly changes.
On the other hand, when we interact with an application that has animations, we can feel the difference. The animations are often subtle. If you aren’t sure what I’m talking about, pay attention the next time you use Slack or Facebook Messenger or iMessage. Airbnb values animation so much that they built Lottie, a library for animation.
In an animated application, the user interface feels alive. When a software team takes the time to build animations into small interactions, the user perceives the animations as polish and attention to detail.
Sarah Drasner has been evangelizing the value of animations for years, and she is an expert at implementing complex and beautiful animations on the web. She works at Microsoft as a developer advocate and joins the show to talk about how to build animations. If you are building a web application and want to create a unique UI, you might find this show useful.
JavaScript supports detailed animations, often through the manipulation of SVG files. SVG stands for “scalable vector graphics” a file format that represents an image in its own DOM. SVG is so flexible because of this DOM format, which defines the different parts of the SVG. This is in contrast to a bitmap, which is just a simple matrix of dots, without any rich metadata.
You could manipulate SVG with raw JavaScript—but most people use a frontend JavaScript framework like React, Angular, or VueJS. Sarah has been implementing most of her recent web animations using Vue, and she is a member of the Vue core team.
Vue has an entertaining story, because it gained popularity in a time when Google was supporting AngularJS and Facebook was supporting ReactJS. The first version of Vue was created from scratch by a single developer, Evan You.
If you are a Vue developer looking for an open source project to hack on, you can check out softwaredaily.com, which is an open source platform to consume content about software. In addition to the Vue web app, we also have the Software Engineering Daily app for iOS and for Android. All of these apps are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help.
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
Are we a media company or a technology company? Facebook and the New York Times are both asking themselves this question.
Facebook originally intended to focus only on building technology–to be a neutral arbiter of information. This has turned out to be impossible. The Facebook newsfeed is defined by algorithms that are only as neutral as the input data. Even if we could agree on a neutral data set to build a neutral newsfeed, the algorithms that generate this news feed are not public, so we have no way to vet their neutrality.
Facebook is such a powerful engine for distribution, it has allowed for a rise in the number of publishers who can get their voice heard. As a result, large media companies have lost market share because Facebook has replaced their distribution.
The New York Times has always been a media company–but the standards for media consumption have shot up. Millions of people produce content for free, and that content is distributed through high quality experiences like Twitter, YouTube, Medium, and Facebook. When a page takes too long to load on NewYorkTimes.com, it doesn’t matter how good the content is–the user is going to navigate away before they read anything.
Today, the New York Times has built out an experienced engineering team. In a previous episode, we reported how the Times uses Kafka to make its old content more accessible. In today’s show, we talk about how the Times uses React and GraphQL to improve the performance and the developer experience of engineers who are building software at the New York Times.
Scott Taylor and James Lawrie are software engineers at the New York Times. In this episode, they explain how the New York Times looks at technology. The user experience on New York Times rivals that of a platform company like Facebook, and this is assisted by technologies originally built at Facebook: React, Relay, and GraphQL.
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
From the publisher's feed