Software Engineering Daily

A data warehouse provides fast access to large data sets for analytics, data science, and dashboards. A data warehouse differs from a transactional database, because you often do not need to update specific records. Because of the read-only nature of the access patterns, and the high volumes of data being queried, the design of a

Mapillary is a platform for gathering photos taken by smartphones and using that data to build a 3D model of the world. Mapillary’s model of the world includes labeled objects such as traffic signs, trees, humans, and buildings. This 3D model can be explored much like you can explore Google Street view. The data set

When Aran Khanna was a college student, he accepted an internship to work at Facebook. Even before his internship started, he started playing around with Facebook’s APIs and applications. Aran built a Chrome extension called Marauder’s Map, which used Facebook Messenger’s web APIs to track where people lived, what their schedule was, and other highly

Airbnb began in 2008 as a monolithic Rails application serving the simple purpose of listing homes for rental. Over time, the number of listings increased dramatically, as did the number of people who were renting. With that scale, the Rails app had to be broken into different services, and entire teams were built out to

We previously released this episode with the wrong audio file and are re-releasing it on a weekend. TIBCO was started in the 90’s with a popular message bus product that was widely used by finance companies, logistics providers, and other systems with high throughput. As TIBCO grew in popularity, the company expanded into other areas

Log messages are fast, high volume, unstructured data. Logs are often the source of metrics, alerts, and dashboards, so these critical systems are downstream from a log management system. A log management system needs to be highly available, so that a failure in one part of your system will not be correlated with failure of

When a database gets large, it can start to perform poorly. This can manifest in slow query speed. You can speed up a query by defining an index, which is a data structure that allows for faster access to the data that is being indexed. As a consequence, whenever you update the database, you will

Linkedin is an organization with thousands of employees. An enterprise of that size starts to develop problems with data collaboration. Data collaboration is the process of sharing and analyzing data with multiple users, such as data scientists, business analysts, and engineers. How do data scientists know what questions to ask? How do business analysts know

Open source software is key to our software infrastructure. Closed source enterprises rely on open source software, but the development processes for closed source and open source software are often different in their approach to continuous integration and delivery. Oren Novotny is a chief architect of DevOps and modern software at BlueMetal Architects where he

GraphQL allows developers to communicate with all of their different data backends through a consistent query interface. A GraphQL query can be translated into queries to MySQL, MongoDB, ElasticSearch, or whatever kind of API or backend is needed to fulfill the GraphQL query. GraphQL users need to set up a GraphQL server to fulfill this