The MapReduce paper was published by Google in 2004. MapReduce is an algorithm that describes how to do large-scale data processing on large clusters of commodity hardware. The MapReduce paper marked the beginning of the “big data” movement. The Hadoop project is an open source implementation of the MapReduce paper. Doug Cutting and Mike Cafarella

An alert is a signal of problematic application behavior. When something unusual happens to your application, an alert can bring that anomaly to your attention. In order to detect unusual events, you need to define the norm. In order to define both normal and problematic behavior, you need metrics. Metrics are measurements of the behavior

A dashboard is a data visualization that aggregates metrics in a way that we can quickly understand. In a modern software company, everyone uses dashboards–from salespeople to DevOps to HR. Each dashboard represents a query that must be updated frequently, so that anyone looking at it is getting up-to-date information. The data set being queried

Moat is one of the most successful advertising technology companies in history. After building a business from measurement of ad impressions, Moat was sold to Oracle for $850 million. Advertising powers the free content on the Internet. Measurement makes it easier for publishers to monetize their content. At Software Engineering Daily, we know this from firsthand experience. The podcast ecosystem has barely any ability to measure success–and that can makeContinue reading...

Most products do not sell themselves. Salespeople bridge the gap between a product creation and a customer who purchases it. People can make a good living on the internet selling niche products–if they can find their customers. The process of taking a large group of potential customers and narrowing it down to only the subset

Querying a search index for objects similar to a given object is a common problem. A user who has just read a great news article might want to read articles similar to it. A user who has just taken a picture of a dog might want to search for dog photos similar to it. In both of these cases, the query object is turned into a vector and compared toContinue reading...

Instacart is a grocery delivery service. Customers log onto the website or mobile app and pick their groceries. Shoppers at the store get those groceries off the shelves. Drivers pick up the groceries and drive them to the customer. This is an infinitely complex set of logistics problems, paired with a rich data set given by the popularity of Instacart. Jeremy Stanley is the VP of data science for Instacart.Continue reading...

A data-driven organization is more efficient because the company can learn what to focus on. In this episode, Edaena Salinas from The Women in Tech Show interviews Rya Sciban, Product Manager at Periscope Data, who explains the needs of data teams in an organization. We talked about what data analysis is and how this changes as the amount of data grows. Rya explained what analytics clusters are and effective ways ofContinue reading...

Different databases have different access patterns. Key-value, document, graph, and columnar databases are useful under different circumstances. For example, if you are a bank, and you have a database of customers and the transactions they have performed, the ideal access pattern for aggregating the total amount of all transactions might be a columnar store. If

With a fast-growing field like data science, it is important to keep some amount of skepticism. Tools can be overhyped, buzzwords can be overemphasized, and people can forget the fundamentals. If you have bad data, you will get bad results in your experimentation. If you don’t know what statistical approach you want to take to