Software Engineering Daily

The intent of the DevOps movement is to get organizations moving faster and more effectively by breaking down siloes, and improving communication. Gene Kim’s book The Phoenix Project illustrated this by telling the fictional story of a company adopting a DevOps mentality. Although that book was fiction, Gene is an experienced engineer, having worked as founder and CTO of Tripwire, a software company that makes security and compliance automation software.

In his new book The DevOps Handbook, Gene presents a practical companion to The Phoenix Project. Together with his co-authors, Gene has written a guide for how to move an organization toward DevOps, and in this episode we explore some of the topics from his book.

Check out an excerpt from Gene’s book here: DevOps Handbook

The post DevOps Handbook with Gene Kim appeared first on Software Engineering Daily.

At Netflix, developers write applications with a variety of requirements–from simple requests for a list of movies to more resource-intensive requests like a complex machine learning workflow. Netflix wants developers to be able to request the resources they need from a compute cluster and receive those resources on-demand, without thinking too much about the state of that pool of resources they are drawing from.

At the cluster level, this means scheduling the application requests intelligently. Sharma Podila, a distributed systems software architect at Netflix, explains how Netflix has built a cloud native scheduling system on top of Mesos.

The post Netflix Scheduling with Sharma Podila appeared first on Software Engineering Daily.

Building a monitoring system is a complex distributed systems problem. Events are produced from different points in an application and must be aggregated in order to form metrics. These events are often ingested by a time series database, which forms the backbone of our monitoring system.

Theo Schlossnagle is the CEO of Circonus, where he has been working on architecting the company’s monitoring software for six years. In this episode, we talk about how to build a monitoring system and the requirements for the underlying time-series database, as well as what monitoring even is.

The post Monitoring Architecture with Theo Schlossnagle appeared first on Software Engineering Daily.

In order to move software updates from the development team to production, companies do a variety of things. Some teams might email files to each other or use FTP or even floppy disks. Most companies today at least use version control systems like Git together with separate servers for development and production. When code is ready to move to production, a build that is on the development server gets copied over to the production servers, and the production servers begin serving real users.

This process is known as deployment, and over the last few decades companies have started deploying more rapidly (even “continuously”), leading to faster iterations and better feedback loops between the software development team and the users of the product. A particularly effective version of this workflow is known as continuous delivery.

In today’s episode, David Rice from ThoughtWorks joins the show to give a short history of continuous delivery, and how continuous delivery actually looks in practice.

The post Continuous Delivery with David Rice appeared first on Software Engineering Daily.

Kafka Streams is a library for building streaming applications that transform input Kafka topics into output Kafka topics. In a time when there are numerous streaming frameworks already out there, why do we need yet another? To quote today’s guest Jay Kreps “the gap we see Kafka Streams filling is less the analytics-focused domain these frameworks focus on and more building core applications and microservices that process data streams.”

Jay is the CEO of Confluent, a company that is building Kafka technology, and he is one of the original authors of Kafka. We began with a brief history of why Kafka evolved to be the message broker of choice for so many data engineering stacks, then we got into a discussion of streaming–what is streaming? How does Kafka communicate with streaming frameworks? Our conversation concluded with a discussion of Kafka Streams and the future of the Kafka data platform.

The post Kafka Streams with Jay Kreps appeared first on Software Engineering Daily.