Software Engineering Daily

When I buy a mattress online, I pay for it with my credit card. Behind the scenes, a complex series of transactions occur between a payment gateway, the credit card company, and a few banks. There are problems with this process–it is slow, complex, and involves the synchronization of several different parties.

Some consumers will not want to purchase the mattress because they do not have cash, and the lending rates they get offered are higher than they are willing to spend. If these consumers were presented with more intelligent loan rates, the lender could still make money, the mattress company could still make money, and the consumer would get a new mattress. It’s a missed opportunity all around.

Affirm is a consumer financial services company. Their first product offers loans to consumers making purchases. In today’s episode, Affirm CTO Libor Michalek explains how Affirm decided to build this product, and what they have done to scale it.

The conversation took me by surprise. Affirm was started by Max Levchin, who was a co-founder of PayPal. I assumed that when Affirm was created, they already knew exactly what they were going to build–because Affirm is a payments company and Max has had knowledge of the payments industry going back two decades. In reality, Affirm started out with more vague ideas around what they were building.

They spent some time running small experiments as they looked for product/market fit–just like a bootstrapped startup would have. It was inspiring to know that even an experienced team is willing to go through the humble process of searching for a product within a space they are deeply familiar with.

We didn’t get to all the questions I was planning to explore, but I hope to do another show about Affirm in the future.

Affirm Careers Page

The post Affirm Engineering with Libor Michalek appeared first on Software Engineering Daily.

Superpedestrian is a robotic bicycle wheel that learns how you pedal and personalizes your bicycle ride. The engineering challenges of Superpedestrian are at the intersection of robotics, software, and real-time analytics. The first half of today’s show is about Superpedestrian. Goss Nuzzo Jones and Matt Cole are engineers at Superpedestrian. The slides for their presentation are also in the show notes.

The second half of today’s show is about HubSpot, a massive business with lots of infrastructure challenges. Thomas Petr explained how HubSpot’s engineering has matured, and some of the scaling problems they have tackled.

Last month, we had three Software Engineering Daily Meetups: in New York, Boston, and Los Angeles. At each of these Meetups, listeners from the SE Daily community got to meet each other and talk about software–what they are building and what they are excited about. I was happy to be in attendance at each of these, and I am posting the talks given by our presenters. The audio quality is not perfect on these, but there are also no ads.

Thank you to HubSpot for hosting this Meetup–they have beautiful offices and if you are looking for a job (or if you want to host a technology Meetup in the Boston area) I strongly recommend checking them out.

We’d love to have you as part of our community. We will have more Meetups eventually, and you can be notified of these by signing up for our newsletter. Come to SoftwareDaily.com and get involved with the discussion of episodes and software projects. You can also check out our open source projects–the mobile apps, and our website.

The post Superpedestrian Robotic Wheel / Infrastructure at HubSpot Meetup Talks appeared first on Software Engineering Daily.

Alexis Le-Quoc started Datadog in 2010, after living through the Internet boom and bust cycle of the late 90s and early 2000s. In 2010, cloud was just starting to become popular. There was a gap in the market for infrastructure monitoring tools, which Alexis helped fill with the first version of Datadog.

Since 2010, the number of different cloud infrastructure products has proliferated. There were new databases, queueing systems, virtualization and containerization tools. Web 2.0 took off, and thousands of new Internet businesses got started. Many of these businesses used Datadog to monitor their increasingly wide range of infrastructure configurations–and Datadog began to scale.

On today’s show, Alexis tells the story of how Datadog grew from its first product into a variety of tools–infrastructure monitoring, logging, and application performance monitoring. Monitoring is a unique challenge–there is a ton of data, the data is latency sensitive, and the data is operationally important. These engineering constraints provide for a great conversation. Alexis is the CTO of Datadog, and we talked about cloud providers, building a business, infrastructure, and how to scale engineering management. Full disclosure: Datadog is a sponsor of Software Engineering Daily.

The post Building Datadog with Alexis Le-Quoc appeared first on Software Engineering Daily.

Google’s central system for managing to compute resources is called Borg. On Borg, millions of Linux containers process a wide variety of workloads. When a new application is spun up, Borg provides that application with the resources it needs.

Workloads at Google usually fall into one of two distinct categories: long-running application workloads (such as Gmail) and batch workloads (such as a MapReduce job). In the early days of Google, the long-lived workloads were scheduled onto a system called “BabySitter” and the batch workloads were scheduled onto a system called “Global Work Queue.”

Borg was the first cluster manager at Google designed to service both long-running and batch workloads from a single system. The second cluster manager at Google was Omega, a project that was created to improve the engineering behind Borg. The innovations of Omega improved the efficiency and architecture of Borg.

More recently, Kubernetes was created as an open source implementation of the ideas pioneered in Borg and Omega. Google has also built a Kubernetes as a service offering that companies use to run their infrastructure in the same way that Google does.

Brian Grant is an engineer at Google who has seen the iteration of all three cluster management systems that have come out of Google. He joins the show to discuss how the workloads at Google have changed over time, and how his perspective on how to build and architect distributed systems has evolved. Full disclosure: Google is a sponsor of Software Engineering Daily.

The post Google Cluster Evolution with Brian Grant appeared first on Software Engineering Daily.

A message broker is an architectural component that sends messages between different nodes in a distributed system.

Message brokers are useful because the sender of a message does not always know who might want to receive that message. Message brokers can be used to implement the “publish/subscribe” pattern, and by centralizing the message workloads within the pub/sub system, it lets system operators scale the performance of the messaging infrastructure by simply scaling that pub/sub system.

Derek Collison has worked on messaging infrastructure for 25 years. He started at TIBCO, then spent time at Google and VMWare. When he was at VMWare, he architected the open source platform Cloud Foundry. While working on Cloud Foundry, Derek developed NATS, a messaging control plane.

Since that time, Derek has started two companies–Apcera and Synadia Communications. In our conversation, Derek and I discussed the history of message brokers, how NATS compares to Kafka and his ideas for how NATS could scale in the future to become something much more than a centralized message bus.

The post NATS Messaging with Derek Collison appeared first on Software Engineering Daily.