Software Engineering Daily

Open source software allows developers to take code from the Internet and modify it for their own use. Open source has allowed innovation to occur on a massive scale. Today, open source software powers our consumer client applications and our backend cloud server infrastructure. 

Linux powers single node operating systems and Kubernetes is the foundation for new distributed systems. Hadoop created an open source distributed file system, Spark gave us a computational runtime on top of it, and Kafka created a middleware platform for shuttling data from one place to another.

There are numerous other examples of how open source has changed the world of software development. Open source has also reshaped the business landscape of infrastructure software companies. 

A common business structure for a modern infrastructure company is the “open core” model. An open core company maintains an open source project that is free to use, but also sells a product or service around that product. Companies with an open core model include Red Hat, HashiCorp, and GitLab.

Many companies are building a thriving business with the open core business model. But these companies do not directly control the most important part of the infrastructure supply chain: the cloud provider.

Cloud providers have a fundamental tension with open core companies because the cloud providers offer services that compete with the open core companies. 

In addition to the issue of cloud providers competing directly with the open core companies, some people have questioned whether Amazon Web Services is capturing an unfair portion of the value that is being created by open source.

Amazon Web Services is the biggest cloud provider, and it has built a large catalog of services that are built off of open source software. But AWS has not historically contributed heavily to open source relative to the value it has captured. 

One example of an open core company which has lost market share to an AWS cloud-hosted offering is Elastic, the open core company which maintains the ElasticSearch open source project. Amazon ElasticSearch Service is a closed-source hosted offering built on top of the ElasticSearch.

Elastic (the company) has increasingly intermingled proprietary software with their open source repository, making it less clear how that open source repository can be used by companies that want to deploy it for their commercial use.

Open core companies such as MongoDB, Redis Labs, and Cockroach Labs have responded to the competitive pressures of AWS by changing their licenses and making it more expensive for cloud providers to offer a cloud-hosted offering of their open source project.

The dynamics between cloud providers and open core companies will continue to evolve in the coming years. The norms around open source are up for debate.

Joseph Jacks is the founder of OSS Capital, a venture firm focused on investments in commercial open source software companies. He returns to the show to discuss the changing landscape of open core companies, and the benefits of permissionless innovation.

The post Permissionless Innovation with Joseph Jacks appeared first on Software Engineering Daily.

React is a set of open source tools for building user interfaces. React was open sourced by Facebook, and includes libraries for creating interfaces on the web (ReactJS) and on mobile devices (React Native).

React was released during a time when there was not a dominant frontend JavaScript library. Backbone, Angular, and other JavaScript frameworks were all popular, but there was not any consolidation across the frontend web development community. Before React came out, frontend developers were fractured into different communities for the different JavaScript frameworks.

After Facebook open sourced React, web developers began to gravitate towards the framework for its one-way data flow and its unconventional style of putting JavaScript and HTML together in a format called JSX. As React has grown in popularity, the React ecosystem has developed network effects. In many cases, the easiest way to build a web application frontend is to compose together open source React components.

After seeing the initial traction, Facebook invested heavily into React, creating entire teams within the company whose goal was to improve React. Dan Abramov works on the React team at Facebook and joins the show to talk about how the React project is managed and his vision for the project.

RECENT UPDATES:

The FindCollabs Open has started. It is our second FindCollabs hackathon, and we are giving away $2500 in prizes. The prizes will be awarded in categories such as machine learning, business plan, music, visual art, and JavaScript. If one of those areas sounds interesting to you, check out findcollabs.com/open!

The FindCollabs Podcast is out!

We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

The post Facebook React with Dan Abramov appeared first on Software Engineering Daily.

RECENT UPDATES:

Podsheets is our open source set of tools for managing podcasts and podcast businesses

New version of Software Daily, our app and ad-free subscription service

Software Daily is looking for help with Android engineering, QA, machine learning, and more

FindCollabs Hackathon has ended–winners will probably be announced by the time this episode airs; we will be announcing our next hackathon in a few weeks, so stay tuned

Open source policy has become a business issue as well as a political one.

Businesses like Elastic, MongoDB (the company), and Redis Labs have started to view the open source licenses of the projects they work on as a means for business defensibility against cloud providers offering similar services. It remains to be seen how viable this strategy will be for the commercial open source vendors.

Companies that do not directly sell commercial open source are also grappling with questions around open source licensing. Facebook has become a force in the open source world through projects like React and GraphQL. Facebook leads these projects, but Facebook is not monetizing them other than to the extent that they use the projects to build Facebook.com.

Facebook’s incentives are aligned with the rest of the industry on the quality of the GraphQL and React projects. Proper licensing can help Facebook keep those incentives in alignment.

Joel Marcey, Michael Cheng, and Kathy Kam from Facebook join me for a discussion of the state of open source licensing, and how that impacts Facebook.

The post Facebook OSS License Policy with Joel Marcey, Michael Cheng, and Kathy Kam appeared first on Software Engineering Daily.

Upcoming events:

A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019

FindCollabs Hackathon at App Academy on April 6, 2019

Data engineering touches every area of an organization.

Engineers need a data platform to build search indexes and microservices. Data scientists need data pipelines to build machine learning models. Business analysts need flexible dashboards to understand the trends and customer use for a product.

Max Beauchemin is a data engineer who has worked at Airbnb, Lyft, and Facebook. He’s the creator of two successful open source projects: Apache Airflow and Apache Superset. In a previous show, Max discussed data engineering at Airbnb, and the usage of Airflow. In today’s show, Max discusses the engineering of Apache Superset.

Superset is an open source business intelligence web application. Superset allows users to create visualizations, slice and dice their data, and query it. Superset integrates with Druid, a database that supports exploratory, OLAP-style workloads.

One reason Superset is distinctive is that it is a full open source application. Many open source projects are tools like databases, command line tools, and web frameworks. Superset is an open source application that can be used by individuals who are not developers–so the audience is wider than the typical open source tool built for engineers.

Max joins the show to talk about his experience as a data engineer at Airbnb and Lyft, and the open source projects he has started.

The post Apache Superset with Maxime Beauchemin appeared first on Software Engineering Daily.

Upcoming events:

A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019

FindCollabs Hackathon at App Academy on April 6, 2019

Red Hat was the first commercial open source software company. For years, investors and entrepreneurs assumed there would never be another Red Hat.

Red Hat’s business was built around enterprise operating system distribution and support. Since the operating system is at the core of how users within a company are doing their job, Red Hat had a lot of leverage and a strong business model. But how many enterprise software products could be so critical to a business that they could manage to offer their software as an open source option yet still make money?

As it turns out, there are many ways to make money in open source.

MySQL ate away at the dominance of Oracle’s database business for similar reasons to Red Hat’s success: much like an operating system, the database layer is critical infrastructure. Cloudera and Hortonworks were able to monetize the open source Hadoop project because Hadoop was hard to deploy and manage.

As cloud infrastructure matured, it became easier to start companies that offered open source software as-a-service. Elastic offers an easy way to use ElasticSearch. RedisLabs offers Redis as a service. MongoDB (the company) offers MongoDB as a service. As it turns out, engineers love to see the source code for databases, but they do not enjoy deploying and managing them and they are happy to pay providers to save them time.

Still, there is continued skepticism of open source businesses.

Today’s debates center around whether individual providers like Elastic can offer a service that competes with an ElasticSearch service offered by AWS. We could just as easily be asking the inverse question– how can AWS compete with an entire company that is dedicated to the deeply technical problem of solving search?

The reality is that most of these open source product categories have an enormous total addressable market, and extremely good unit economics. This applies to both cloud providers and point solution providers. Investors often talk about how much they love subscription businesses. When a company starts purchasing infrastructure-as-a-service from you, it is like they are buying a subscription where the annuity increases over time!

When an investor says they are worried about a giant cloud provider offering the same service as an open source company, it is similar to the investor being worried that a new sales tool is going to be duplicated by Salesforce. The market always needs new sales tools–and the market needs those tools to be offered both by Salesforce and by smaller CRM companies.

In the world of commercial open source, there is plenty of room for both point solution providers and cloud providers. But they are competing for the same customers, and the competitive battlefield is expanding to the nuanced world of software licensing. By changing their licenses, open source projects like Kafka, MongoDB, and Redis can prohibit AWS from certain usage patterns. This might offer some protection for companies based around the point solutions–companies like Confluent and RedisLabs.

Beyond the fracas of the battle between cloud providers and point solutions, there are newer open source companies with models that do not fit tightly into any historical business models. HashiCorp makes a suite of differentiated open source tools that have not been seriously contested or offered as a service by cloud providers. GitLab makes an open source platform that is built with monitoring, logging, CI, and code hosting out of the box.

As the world of open source business models expands, more companies will find opportunity in open sourcing the code that runs their products. In many cases, they will find that it strengthens their advantage rather than weakens it. The defensibility of many businesses relies more on data and network effects than the contents of the codebase. We may see the default question gradually shift from “why should I open source my codebase?” to “why shouldn’t I open source my codebase?”

Mike Volpi is a partner at Index Ventures and has invested in many open source businesses over the last decade. He is on the board of Confluent, Cockroach Labs, Kong, and Elastic. Mike joins the show to share his perspective on open source business models of the past, present, and future.

The post OSS Businesses with Mike Volpi appeared first on Software Engineering Daily.

GitLab is an open source platform for software development.

GitLab started with the ability to manage git repositories and now has functionality for collaboration, issue tracking, continuous integration, logging, and tracing. GitLab’s core business is selling to enterprises who want a self-hosted git installation, such as banks or other companies who prefer not to use a git service in the cloud.

The vision for GitLab is to provide a platform for managing the full software development lifecycle, from code hosting to deployment–as well as tools for observability and project management.

Sid Sijbrandij is the CEO of GitLab and he joins the show to talk about the product, the business, and the company’s vision for the future. GitLab’s strategy is to offer a set of tools that work for developers out of the box, cutting down on time spent integrating each individual vendor.

The post GitLab with Sid Sijbrandij appeared first on Software Engineering Daily.

An operating system kernel manages the system resources that are needed to run applications. The Linux kernel runs most of the smart devices that we interact with, and is the largest open source project in history.

Shuah Khan has worked on operating systems for two decades, including 13 years at HP and 5 years at Samsung. She has worked on proprietary operating systems and a variety of Linux operating system environments, including mobile devices. Shuah joins the show to discuss her work within Linux and her experience contributing to open source.

Shuah has made significant contributions to kselftest, a set of tests for Linux. Testing the Linux kernel is complicated. Because there is so much depth to the codebase, and such a variety of ways that Linux can be used, there is also a variety of ways that the operating system gets tested. There is smoke testing, performance testing, and regression testing. There are trees of tests, and as a developer you may only want to run a subset of the tests in that tree.

The conversation with Shuah ranged from the low level practices of testing the kernel to a high level discussion of how the Linux kernel can reveal dynamics of human nature.

The post Linux Kernel Development with Shuah Khan appeared first on Software Engineering Daily.

Edge computing refers to computation involving drones, connected cars, smart factories, or IoT sensors. Any software deployment that is not a large centralized server installation could qualify as an edge device–even a smartphone.

Today, much of our heavy computation takes place in the cloud–a set of remote data centers some distance away from our client devices. For many use cases, this works fine. But there are a growing number of use cases with lower latency and higher bandwidth requirements at the edge.

A simple example is video. Let’s say you want to record a video stream, and detect people in that video stream in real time. Based on who those people in the video stream are, you want to do different things–maybe you want to send them a text message, or report to the police that a dangerous person has entered the premises. This video stream could be captured by a drone, or by a smart car, or by a video camera mounted somewhere.

Where is the video stream getting stored? Where is the machine learning model running? How do you deploy new machine learning models to the operating system with the machine learning model? This is a simple example, and there are many open questions as to how to best solve such a problem.

With the increased resource constraints at the edge, there is a need for new hardware and software to power these edge applications. This led to the creation of LF Edge, a new open source group under the Linux Foundation. The goal of LF Edge is to build an open source framework for the edge.

Arpit Joshipura is the general manager of networking, orchestration, edge computing, and IoT with the Linux Foundation. He joins the show to describe the state of edge computation, and the mission of LF Edge.

This episode was exciting for several reasons. After seeing the rise of Kubernetes for container orchestration, we know that a popular open source technology that solves a widespread problem can have dramatic influence on the software world. And when multiple large companies get involved in that open source project, it can gain traction quite quickly.

Edge computing has a large set of unanswered questions, but telecom providers like AT&T and large infrastructure companies like Dell EMC are getting heavily involved with the Linux Foundation Edge group. This represents a significant expansion of the open source model, and a suggestion of further investment into open source projects in the near future.

The post Edge Computing Open Source with Arpit Joshipura appeared first on Software Engineering Daily.

When a user makes a request to product like The New York Times, that request hits an API gateway. An API gateway is the entry point for an external request. An API gateway serves several purposes: authentication, security, routing, load balancing, and logging.

API gateways have grown in popularity as applications have become more distributed, and companies offer a wider variety of services. If an API is public, and anyone can access it, you might need to apply rate limiting so that users cannot spam the API. If the API is private, the user needs to be authenticated before the request is fulfilled.

Kong is a company that builds infrastructure for API management. The Kong API gateway is a widely used open source project, and Kong is a company built around supporting and building on top of the API gateway.

Marco Palladino is the co-founder and CTO of Kong. He joins the show to tell the story of starting Kong eight years ago, and how the API gateway product evolved out of an API marketplace. Marco also discusses the architecture of Kong and his vision for how the product will develop in the future–including the Kong service mesh.

The post Kong API Platform with Marco Palladino appeared first on Software Engineering Daily.

Building software was simplified by cloud providers. With the cloud, it became much easier to deploy a server, spin up a database, and scale an application. Cloud providers like AWS gave developers access to these infrastructure primitives like storage and compute.

On top of those primitives, numerous API companies have been built. An API company offers a more specific set of services. Twilio offers SMS text messaging API services. Stripe offers payment API services. These APIs give developers another level of tooling to build software out of.

Developers can now think of entire applications in terms of APIs, and the number of APIs is growing rapidly. From business services such as booking a flight to machine learning models like image classification, the “API economy” has given developers a huge catalog of tools.

Since developers have this additional leverage, software can be built with smaller teams. The codebase can also be smaller. But one area where the complexity is growing is the number of APIs that need to be managed. For each API, there is a different system for integrating the API into your application. Different API providers have different levels of reliability.

Another area of difficulty is the discoverability of APIs. If I don’t know about a flight search API, I am never going to think of what applications I could build on top of that. There are APIs for generating memes, and APIs for easily querying what music is trending across the world.

RapidAPI is a marketplace for APIs. It includes search and discovery features for the wide variety of different APIs that can be found across the internet. RapidAPI is also a system for integrating with multiple APIs through it’s API management system.

Iddo Gino is the CEO and founder of RapidAPI, and he joins the show to discuss the motivation for creating an API marketplace, as well as the engineering behind RapidAPI.

The post RapidAPI: API Marketplace with Iddo Gino appeared first on Software Engineering Daily.