
Sign up to save your podcasts
Or


Facebook has released open source software projects that have changed the industry. The most impactful projects to date are the React frontend user interface tools: ReactJS and React Native.
Before React became popular, there were multiple competing solutions for the dominant frontend JavaScript framework. React became the most prominent because of its invention of JSX, its one-way data flow, and the strength of its community.
The React community is led by Facebook engineers. Facebook has a full-time team dedicated to improving React. Facebook also has the benefit of seeing the hardest problems in web development ahead of the rest of the industry, since Facebook is always pushing the leading edge of what software can do.
As React evolves, Facebook itself is the first user of updates to the libraries. Engineers on the React team will proactively update the applications on other teams with new React code. In this way, Facebook’s React team is similar to a platform engineering team, except the open source community benefits from the improvements as well.
Tom Occhino is an engineering director at Facebook. He joins the show to discuss his ten years of experience at the company and his role today as a senior leader in charge of open source developers.
Facebook has released open source software projects that have changed the industry. The most impactful projects to date are the React frontend user interface tools: ReactJS and React Native.
Before React became popular, there were multiple competing solutions for the dominant frontend JavaScript framework. React became the most prominent because of its invention of JSX, its one-way data flow, and the strength of its community.
The React community is led by Facebook engineers. Facebook has a full-time team dedicated to improving React. Facebook also has the benefit of seeing the hardest problems in web development ahead of the rest of the industry, since Facebook is always pushing the leading edge of what software can do.
As React evolves, Facebook itself is the first user of updates to the libraries. Engineers on the React team will proactively update the applications on other teams with new React code. In this way, Facebook’s React team is similar to a platform engineering team, except the open source community benefits from the improvements as well.
Tom Occhino is an engineering director at Facebook. He joins the show to discuss his ten years of experience at the company and his role today as a senior leader in charge of open source developers.
Facebook generates high volumes of data at a rapid pace.
Dhruba Borthakur joined Facebook in 2008 to work on data infrastructure. His early projects at Facebook were around Hadoop, the distributed file system and MapReduce computation platform that laid the foundation for the “big data” movement.
At the time, Facebook was generating as much data as any other startup, and the company needed to stay at the leading edge of scalability techniques for its Hadoop Distributed File System (HDFS) cluster.
Traditionally, Hadoop managed its file system by synchronizing the coordination of the different data nodes with the help of a single master node. At Facebook, the scale of the data was such that the HDFS cluster had thousands of data nodes, which was too much volume for a single master node to handle. Dhruba helped implement redundancy at the master node to create a more resilient system.
The early days of the big data movement was focused on batch processing. A company like Facebook would gather large amounts of data into databases and HDFS, and run offline analytics workloads to gather reports on an hourly, daily, or weekly basis.
Over time, data infrastructure has moved closer to a “real time” processing model. Data infrastructure does not only support batch offline reporting–it also supports machine learning jobs that need to be run on a more frequent basis. These jobs have lower latency requirements, and have driven the adoption of in-memory stream processing systems like Spark and Flink.
Dhruba joins the show to discuss his time at Facebook building data infrastructure. He takes us through the major projects he worked on, including the early Hadoop infrastructure, the refactoring of online user workloads to be more “pull” based than “push” based, and the creation of RocksDB, a storage engine he helped create at Facebook.
Today, Dhruba is the CTO and co-founder of Rockset, a company that builds data infrastructure and database APIs on top of RocksDB. Rockset is building infrastructure for modern technology companies–many of which are facing problems that bear significant resemblance to the ones Facebook encountered as it scaled.
Facebook generates high volumes of data at a rapid pace.
Dhruba Borthakur joined Facebook in 2008 to work on data infrastructure. His early projects at Facebook were around Hadoop, the distributed file system and MapReduce computation platform that laid the foundation for the “big data” movement.
At the time, Facebook was generating as much data as any other startup, and the company needed to stay at the leading edge of scalability techniques for its Hadoop Distributed File System (HDFS) cluster.
Traditionally, Hadoop managed its file system by synchronizing the coordination of the different data nodes with the help of a single master node. At Facebook, the scale of the data was such that the HDFS cluster had thousands of data nodes, which was too much volume for a single master node to handle. Dhruba helped implement redundancy at the master node to create a more resilient system.
The early days of the big data movement was focused on batch processing. A company like Facebook would gather large amounts of data into databases and HDFS, and run offline analytics workloads to gather reports on an hourly, daily, or weekly basis.
Over time, data infrastructure has moved closer to a “real time” processing model. Data infrastructure does not only support batch offline reporting–it also supports machine learning jobs that need to be run on a more frequent basis. These jobs have lower latency requirements, and have driven the adoption of in-memory stream processing systems like Spark and Flink.
Dhruba joins the show to discuss his time at Facebook building data infrastructure. He takes us through the major projects he worked on, including the early Hadoop infrastructure, the refactoring of online user workloads to be more “pull” based than “push” based, and the creation of RocksDB, a storage engine he helped create at Facebook.
Today, Dhruba is the CTO and co-founder of Rockset, a company that builds data infrastructure and database APIs on top of RocksDB. Rockset is building infrastructure for modern technology companies–many of which are facing problems that bear significant resemblance to the ones Facebook encountered as it scaled.
Facebook moves fast because of vision, collaboration, and trust. The fast pace of development is enabled by constantly improving infrastructure and a sense of unity throughout the company.
In Facebook’s early days, there was an emphasis on rapidly deploying new code to drive constant improvement and experimentation.
Product quality was maintained by engineers closely checking each other’s code reviews, rather than writing detailed unit testing suites. Facebook engineers had a sense for how the product should operate, and they were able to evaluate whether a new feature was working properly by testing a live version of that feature.
At Facebook, the vision of the company is clearly communicated to the employees. Every employee within Facebook can articulate the vision for the company, and will use similar language in describing that vision.
Since the employees are aligned on strategy, they can also align in their implementation of product features. This reduces conflicts across roles and between teams.
Facebook has also shown a willingness to trust its engineers. Trust was exemplified by Facebook’s tolerance for failures in the early days. When an engineer broke a build, or shipped a feature that failed to gain traction, that engineer was usually not punished. They may have even been rewarded, if the company could learn significantly from such an error.
Raylene Yung was an engineer at Facebook from 2009 to 2015. As she moved from individual contributor to manager to engineering director, Raylene worked on products including News Feed, Timeline, Privacy, and Sharing.
Raylene joins the show to give her reflections on the Facebook product and engineering environment. She explained how Facebook’s culture of collaboration, vision, and trust drive fast product development and minimizes conflict.
Raylene left Facebook and joined Stripe, where she worked on payments systems and international expansion for almost four years.
Facebook moves fast because of vision, collaboration, and trust. The fast pace of development is enabled by constantly improving infrastructure and a sense of unity throughout the company.
In Facebook’s early days, there was an emphasis on rapidly deploying new code to drive constant improvement and experimentation.
Product quality was maintained by engineers closely checking each other’s code reviews, rather than writing detailed unit testing suites. Facebook engineers had a sense for how the product should operate, and they were able to evaluate whether a new feature was working properly by testing a live version of that feature.
At Facebook, the vision of the company is clearly communicated to the employees. Every employee within Facebook can articulate the vision for the company, and will use similar language in describing that vision.
Since the employees are aligned on strategy, they can also align in their implementation of product features. This reduces conflicts across roles and between teams.
Facebook has also shown a willingness to trust its engineers. Trust was exemplified by Facebook’s tolerance for failures in the early days. When an engineer broke a build, or shipped a feature that failed to gain traction, that engineer was usually not punished. They may have even been rewarded, if the company could learn significantly from such an error.
Raylene Yung was an engineer at Facebook from 2009 to 2015. As she moved from individual contributor to manager to engineering director, Raylene worked on products including News Feed, Timeline, Privacy, and Sharing.
Raylene joins the show to give her reflections on the Facebook product and engineering environment. She explained how Facebook’s culture of collaboration, vision, and trust drive fast product development and minimizes conflict.
Raylene left Facebook and joined Stripe, where she worked on payments systems and international expansion for almost four years.
Facebook was built using PHP, a programming language that was used widely in the late 90s and early 2000s. PHP allows developers to get web applications built quickly and easily, although PHP has a reputation for being difficult to scale.
In the early days of Facebook, the company was scaling rapidly on every dimension.
New users were piling into Facebook. Existing users were increasing their interactions and developing new patterns of usage. The Facebook application was rolling out new features quickly, adding them into the Facebook PHP codebase.
A common pattern for scaling a large software application is to use a microservices architecture, breaking up the monolithic application into small services which can scale independently. For many applications, this pattern works well. But for some applications, microservices makes less sense.
Microsoft Excel is one example. In Excel, a user is making updates to a complex data model using formulas, functions, and other in-app tools that need to be fast, performant, and integrated. The user needs to have a sense that the Excel data model will update quickly in response to changes.
A software team working on a spreadsheet product such as Excel might prefer to keep all the application logic in a monolithic application.
A monolith can centralize logic and make it easier to reason about. A monolith can reduce the number of network hops, cutting down on distributed systems problems. Testing and deploying a monolithic application can be less complex than doing so in a distributed, microservices system.
Facebook chose to scale its PHP monolith rather than breaking it up into distributed microservices. Scaling PHP allowed Facebook to continue moving fast without going through a painful refactoring that would have slowed down the entire company.
The first effort to scale PHP involved transpiling the entire PHP application into C++. This C++ version of Facebook ran faster and with a lower memory footprint. But C++ required ahead-of-time compilation: the PHP codebase had to be converted to C++ in one synchronous step.
The Hip Hop Virtual Machine (HHVM) was started to allow the Facebook PHP monolith to run on the V8 JavaScript virtual machine.
The advantage of running PHP on V8 was that it allowed Facebook to transpile PHP to JavaScript, an interpreted language that fit better with Facebook’s dynamic application build path. HHVM also allows Facebook to take advantage of the rapidly improving JavaScript runtime environment.
Keith Adams was an engineer at Facebook for six years, where he helped develop infrastructure to scale PHP effectively. Keith is now the chief architect at Slack, which is also a scaled PHP application. Keith returns to Software Engineering Daily to discuss why and how Facebook scaled PHP.
Data engineering allows a company to take advantage of the large quantities of data that the company has generated. In many companies, new data has been produced rapidly for many years, but the company has not been able to take full advantage of it.
Creating large data sets does not provide immediate value for a company. A company needs to perform data engineering and data science to take full advantage of it.
When data gets generated, it is stored in a database, data lake, or API backend like Google Analytics. In order to manipulate that data, it is often pulled into a data warehouse. A data warehouse provides fast access time to large quantities of data.
Pulling data from a source like a database or data lake into a data warehouse requires a process known as extract and load. Once the data is in the data warehouse, it may also undergo a transform, which enriches the data or puts it in a format that is easier to make use of. Once data is in a data warehouse, it can be used to build models, interactive dashboards, and Jupyter Notebooks.
The data engineering lifecycle has many different components, which is why data engineering can often be intimidating to a company that is trying to make use of their data. Meltano is a project with the goal of providing a system of conventions for managing the data engineering lifecycle. Meltano was started by GitLab, and the Meltano project has some strategic similarities to GitLab.
Danielle Morill is the general manager of Meltano at GitLab. She joins the show to discuss the world of data engineering, and the architecture of Meltano. We touch on the different components of a data engineering pipeline, and the most acute pain points for data engineers.
Software engineers have a wide variety of media to choose from, including podcasts, blogs, YouTube videos, conferences. The amount of software engineering media that is available is growing and accelerating.
Eight years ago, there were not as many options for information about software. Charles Max Wood founded Devchat.tv to create a network of podcasts and other content for software engineers. Today, his podcasts include the popular shows JavaScript Jabber, Ruby Rogues, and Adventures in Angular.
Chuck joins the show for a conversation about software media. This was his second time on the show, with his first episode exploring his podcast JavaScript Jabber.
Software engineers have a wide variety of media to choose from, including podcasts, blogs, YouTube videos, conferences. The amount of software engineering media that is available is growing and accelerating.
Eight years ago, there were not as many options for information about software. Charles Max Wood founded Devchat.tv to create a network of podcasts and other content for software engineers. Today, his podcasts include the popular shows JavaScript Jabber, Ruby Rogues, and Adventures in Angular.
Chuck joins the show for a conversation about software media. This was his second time on the show, with his first episode exploring his podcast JavaScript Jabber.
From the publisher's feed