
Sign up to save your podcasts
Or


Large software companies have lots of users, and the activity from those users results in high volumes of traffic. These companies also have a large surface area across the enterprise.
There are hundreds of services and databases that are fulfilling user requests. As these requests enter the infrastructure of the enterprise, the requests travel through the different services and result in database queries, payments, and other transactions. These transactions result in the generation of log messages. The log messages tell the story of what is happening across the entire company.
Log messages can provide valuable data for security and site reliability engineering. But analyzing a high volume of log data requires a scalable system that can account for that high volume.
Jack Naglieri is the CEO of Panther Security. He previously worked at Airbnb, where he helped develop a system called StreamAlert. At Airbnb, log messages are buffered into distributed queueing systems like Kafka or Kinesis, and they are written to bucket storage systems like S3. Those logs are processed by AWS Lambda functions that test the log messages for rules defined by a system operator.
Jack left Airbnb and started Panther Security to generalize the tools he built within Airbnb and build a company around the same ideas. Jack joins the show to discuss modern logging infrastructure, his work at Airbnb, and his experience building Panther.
Distributed systems are required to run most modern enterprise software. Application services need multiple instances for scalability and failover. Large databases are sharded onto multiple nodes. Logging services, streaming frameworks, and continuous integration tools all require the orchestration of more than one server.
Deploying a distributed system has historically been difficult because the nodes of the system must be managed by the underlying infrastructure. If I have a distributed database that I want to deploy, the complexity of that deployment is going to be different depending on whether I am running on AWS, or VMware, or my own bare metal server infrastructure.
Heterogeneous server infrastructure makes it hard to sell distributed applications that get deployed to that infrastructure. A vendor that is selling a distributed database would need to figure out how to make their database work on the infrastructure of any given customer.
Kubernetes has simplified the process of deploying a distributed application. Kubernetes is a container orchestration system that has steadily grown in popularity, to the point where the ecosystem is mature and the software is stable. Now that the software industry has a reliable, portable means of deploying a distributed application, the enterprise software market is becoming easier to enter for companies that sell a distributed application.
Replicated is a company that builds products for software delivery. Replicated allows for the distribution and updating of applications that would have been hard to deploy in the past. Grant Miller and Marc Campbell are the CEO and CTO of Replicated, and they join the show to talk about the modern enterprise software market, and the process of delivering software to companies that might otherwise have trouble consuming it. Full disclosure: Replicated is a sponsor of Software Engineering Daily.
Distributed systems are required to run most modern enterprise software. Application services need multiple instances for scalability and failover. Large databases are sharded onto multiple nodes. Logging services, streaming frameworks, and continuous integration tools all require the orchestration of more than one server.
Deploying a distributed system has historically been difficult because the nodes of the system must be managed by the underlying infrastructure. If I have a distributed database that I want to deploy, the complexity of that deployment is going to be different depending on whether I am running on AWS, or VMware, or my own bare metal server infrastructure.
Heterogeneous server infrastructure makes it hard to sell distributed applications that get deployed to that infrastructure. A vendor that is selling a distributed database would need to figure out how to make their database work on the infrastructure of any given customer.
Kubernetes has simplified the process of deploying a distributed application. Kubernetes is a container orchestration system that has steadily grown in popularity, to the point where the ecosystem is mature and the software is stable. Now that the software industry has a reliable, portable means of deploying a distributed application, the enterprise software market is becoming easier to enter for companies that sell a distributed application.
Replicated is a company that builds products for software delivery. Replicated allows for the distribution and updating of applications that would have been hard to deploy in the past. Grant Miller and Marc Campbell are the CEO and CTO of Replicated, and they join the show to talk about the modern enterprise software market, and the process of delivering software to companies that might otherwise have trouble consuming it. Full disclosure: Replicated is a sponsor of Software Engineering Daily.
Chat systems have been a part of software development for decades. Older systems like Pidgin and Yammer were surpassed by newer systems like HipChat. And when Slack was created, it quickly became a part of most software companies. But Slack does not fulfill the needs of every company.
Mattermost is an open-source chat system. Mattermost can be configured to work within enterprises that have strong constraints around compliance and data governance. Whereas Slack is a SaaS product that requires users to send their data to the cloud servers managed by Slack, Mattermost allows the enterprise to decide how data moves through services, and where the databases are hosted.
Ian Tien is the CEO of Mattermost, and he joins the show to talk about why many companies need their chat system to be hosted in a private cloud or on-premises. In a previous episode with CTO Corey Hulen we discussed the engineering behind the company. In today’s episode, we explore the management and strategy of the business, as well as some additional engineering, since Ian Tien’s background is as a software engineer and computer scientist.
Chat systems have been a part of software development for decades. Older systems like Pidgin and Yammer were surpassed by newer systems like HipChat. And when Slack was created, it quickly became a part of most software companies. But Slack does not fulfill the needs of every company.
Mattermost is an open-source chat system. Mattermost can be configured to work within enterprises that have strong constraints around compliance and data governance. Whereas Slack is a SaaS product that requires users to send their data to the cloud servers managed by Slack, Mattermost allows the enterprise to decide how data moves through services, and where the databases are hosted.
Ian Tien is the CEO of Mattermost, and he joins the show to talk about why many companies need their chat system to be hosted in a private cloud or on-premises. In a previous episode with CTO Corey Hulen we discussed the engineering behind the company. In today’s episode, we explore the management and strategy of the business, as well as some additional engineering, since Ian Tien’s background is as a software engineer and computer scientist.
The word “DevOps” has a different definition depending on who you ask.
For some people, it is about the process of managing and releasing code. It can involve container management and server orchestration. It can involve infrastructure-as-code, and safer configuration management. In addition to a set of technologies, DevOps can be seen as a management concept that describes agile practices, and breaking down communication barriers between different teams.
One thing that most software companies have decided is that whatever DevOps is, we want it. We want to release more software, we want to do it faster, and we want to do it safer. We want streamlined communication between management and engineering. We want a full understanding of the “value chain” of software.
Despite the elusiveness of a single description for what DevOps is, GitLab can credibly describe itself as a tool that satisfies the DevOps needs of most enterprises. GitLab started as an open source version control management system based on Git. It has expanded into products that include continuous integration, security, issue tracking, and monitoring.
The trajectory of GitLab into such a large platform is something that nobody anticipated. The best explanation for how it happened is that it is the downstream result of an engineer within GitLab deciding that the code hosting product needed to have a continuous integration product bundled with it as an option for a tightly coupled, unified workflow.
Today, there are many enterprises trying to make a big set of changes to their development practices. The world is consolidating around Git for version control and Kubernetes for container management. Almost every enterprise is figuring out a “cloud strategy”. Every team wants to have continuous integration, and they want it to have some security products paired with that release workflow in a popular, vaguely defined set of practices known as “DevSecOps”.
With so many changes coming to enterprises, it turns out that many of these enterprises just want some sane defaults. When GitLab came to market with a bundled CI and code hosting product, the company discovered that the customers were very happy to have integrated tools that worked well out of the box. This was in stark contrast to the years of NxN tooling integration that an enterprise would have to make to stitch together their broad range of carefully selected tools.
Sid Sibrandij is the CEO of GitLab, and he joins the show for a conversation about how GitLab arrived at its product development strategy. In a previous episode, Sid discussed some of the core features and history of GitLab. Today’s show expands on many of the subjects we explored previously. We also had a spirited discussion of the modern nature of work, and how GitLab’s unique culture and fully remote team have evolved as the company has scaled.
The ridesharing infrastructure of Lyft has a high volume of traffic that is mostly handled by servers on AWS. When Vicki Cheung joined Lyft in 2018, the company was managing containers with an internally built container scheduler. One of her primary goals at the company was to move Lyft to Kubernetes.
In today’s episode, Vicki gives an overview of Lyft infrastructure and the core engineering problems within the company. One subject she touched on was the network communications between the user on a mobile phone and the cloud backend. This was a topic we explored in detail on a previous episode about Envoy Mobile with Matt Klein.
Vicki also discussed the broader Kubernetes ecosystem, as well as her time at OpenAI, where she managed infrastructure deployments for scheduling large machine learning jobs.
If the Internet was reimagined with the software and hardware infrastructure we have today, what would it look like?
That is the question that DFINITY is working on answering. DFINITY’s goal is to build a decentralized, secure Internet computer. DFINITY takes concepts from the cryptocurrency world, but it is focused on computation, not financial products. DFINITY can be thought of as a decentralized cloud provider, with redundancy and scalability properties that are achieved by operating on data centers across the world.
DFINITY wants to host web applications such as the ones that we use today on centralized servers. A developer who wants to run their application on DFINITY compiles their code to WebAssembly and deploys it to the DFINITY decentralized runtime. Transactions across DFINITY applications are processed through a collateralized proof-of-stake system to ensure reliable, decentralized computation.
DFINITY is an ambitious project, and it would seem nearly impossible to bring to market if not for the quality of the team. DFINITY has hired Andreas Rossburg, a co-designer of WebAssembly, as well as talented engineers across security, web development, and backend infrastructure.
Dominic Williams is the president and chief scientist of DFINITY, and he joins the show to talk about the vision for DFINITY and the roadmap to making it a reality.
Webflow is a visual programming tool used by designers, developers, and other technical users. Webflow is a leader in the “low code” or “no code” category of software tools that has become prominent in the last few years.
Webflow has been years in the making. In a previous show with Webflow CEO Vlad Magdalin, he told the story of being heads down on Webflow, steadily working through the engineering problems that stood between him and his vision of a visual programming environment.
In the early days, it was unclear who would even want to use Webflow. A critic of Webflow might have said that Webflow was too high level for developers, and too technical for designers. But Webflow caters to a large subset of developers and designers who want the kind of low code experience that Webflow provides. The product has also helped define a new category of knowledge worker: the “visual programmer.”
Bryant Chou is a co-founder of Webflow and was the CTO in the early days. He joins the show to discuss the engineering problems that the company has had to work through, and his perspective on how Webflow fits into the software market going forward.
Webflow is a visual programming tool used by designers, developers, and other technical users. Webflow is a leader in the “low code” or “no code” category of software tools that has become prominent in the last few years.
Webflow has been years in the making. In a previous show with Webflow CEO Vlad Magdalin, he told the story of being heads down on Webflow, steadily working through the engineering problems that stood between him and his vision of a visual programming environment.
In the early days, it was unclear who would even want to use Webflow. A critic of Webflow might have said that Webflow was too high level for developers, and too technical for designers. But Webflow caters to a large subset of developers and designers who want the kind of low code experience that Webflow provides. The product has also helped define a new category of knowledge worker: the “visual programmer.”
Bryant Chou is a co-founder of Webflow and was the CTO in the early days. He joins the show to discuss the engineering problems that the company has had to work through, and his perspective on how Webflow fits into the software market going forward.
From the publisher's feed