“Be aggressive in vision, but conservative in operation.” Uber is a transportation company with a high volume of temporal spacial data, constantly being collected from the devices of its users. At any given time, the engineers and data scientists at Uber need to be able to query the system, and understand what is going on

“I’m always worried that if you teach too much magic, people don’t learn the basics – they don’t know why something is working, they just know the documentation said it should work that way.” On Software Engineering Daily, we often discuss big data in terms of data engineering and data science. Data engineering is the

“The world is becoming more and more interactive, and people want answers right away, so you’re seeing the rise of stream processing and real-time.” Big data is yesterday–fast data is now. FiloDB is a reactive columnar OLAP database that is built on Cassandra and Spark. Today’s guest is Evan Chan, creator of FiloDB. In our

“There isn’t any central node in Cassandra. Every node is a peer, there is no master – there is no single point of failure.” Apache Cassandra can serve as both the real-time data store for online transactional applications, as well as the read-intensive database for data warehousing operations. In order to combine these two use

“HDFS is going to be a cockroach – I don’t think its ever going away.” Hadoop was created in 2003. In the early years, Hadoop provided large scale data processing with MapReduce, and distributed fault-tolerant storage with the Hadoop Distributed File System. Over the last decade, Hadoop has evolved rapidly, with the support of a

“One big transformation we’re seeing right now is the slow agonizing death of MapReduce.” When a company gets big enough, there is so much data to be processed that an entire data engineering team becomes responsible for managing this data and making it available to other teams. Airbnb is one such company. Max Beauchemin works

“You want to take a scientist who knows a little bit of matlab programming and try to teach them mapreduce, and write a mapreduce program in java to do image processing? It’s a disaster!” Apache Spark is replacing MATLAB in the domain of computational neuroscience. The constraints of running MATLAB on a single machine can’t

“There’s a lot of value in moving logic to the data rather than moving data to the logic. And the issue here is the data is a lot bigger than the logic.” NewSQL is a class of modern relational databases that seek to provide the same scalable performance of NoSQL systems for OLTP, while still

This episode is different from the traditional interview format of Software Engineering Daily, and focuses on the history of Hadoop. Thanks to Marco Bonaci for allowing us to republish this in audio format. You can find the original post here: History of Hadoop If you like this podcast, check out Marko’s book Spark in Action (affiliate

“Benchmarks are all crap, but there are some benchmarks that are better than others.”

Continue reading…