Hadoop Distributed File System

The amount of data that can be generated and stored in academic and industrial projects and applications is increasing rapidly. Big data analytics technologies have established themselves as a solution for big data challenges to the scalability problems of traditional database systems. The vast amounts of new data that is collected, however, usually is not as easily analyzed as curated, structured data in a data warehouse is. Typically, these data are noisy, of varying format and velocity, and need to be analyzed with techniques from statistics and machine learning rather than pure SQL-like aggregations and drill-downs. Moreover, the results of the analyses frequently are models that are used for decision making and prediction. The complete process of big data analysis is described as a pipeline, which includes data recording, cleaning, integration, modeling, and interpretation.

In this lecture, we will discuss big data systems, i.e., infrastructures that are used to handle all steps in typical big data processing pipelines. We will learn about data center infrastructure and scale-out software systems. The software discussed will cover the full big data stack, i.e., distributed file systems, Map Reduce, key value stores, stream processing, graph processing, ML systems.

Education

Courses

The amount of data that can be generated and stored in academic and industrial projects and applications is increasing rapidly. Big data analytics technologies have established themselves as a solution for big data challenges to the scalability problems of traditional database systems. The vast amounts of new data that is collected, however, usually is not as easily analyzed as curated, structured data in a data warehouse is. Typically, these data are noisy, of varying format and velocity, and need to be analyzed with techniques from statistics and machine learning rather than pure SQL-like aggregations and drill-downs. Moreover, the results of the analyses frequently are models that are used for decision making and prediction. The complete process of big data analysis is described as a pipeline, which includes data recording, cleaning, integration, modeling, and interpretation. In this lecture, we will discuss big data systems, i.e., infrastructures that are used to handle all steps in typical big data processing pipelines. We will learn about data center infrastructure and scale-out software systems. The software discussed will cover the full big data stack, i.e., distributed file systems, Map Reduce, key value stores, stream processing, graph processing, ML systems.

Share Hadoop Distributed File System

Sign up to save your podcasts

Hadoop Distributed File System

Hadoop Distributed File System

More shows like Big Data Systems (WT 2020/21) - tele-TASK

Design Thinking for Digital Engineering (SS 2018) - tele-TASK

Einführung in die Programmiertechnik II (SS 2018) - tele-TASK

Betriebssysteme II (SS 2018) - tele-TASK

Internet Security (WT 2018/19) - tele-TASK

Distributed Data Management (WT 2018/19) - tele-TASK

Neurodesign Lecture – Physiological Perspectives on Engineering Design, Creativity, Collaboration and Innovation (WT 2019/20) - tele-TASK

Neurodesign Lecture - Artificial Intelligence and the Neuroscience of Creativity (WT 2020/21) - tele-TASK

Neurodesign Lecture - Designing for Empathy in Business Contexts (Wintersemester 2021/2022) - tele-TASK

Betriebssysteme I (WS 2021/22) - tele-TASK

Ringvorlesung - Database Research (WT 2021/22) - tele-TASK