Is ETL dead in Data Science and Big Data?

In today's podcast I share with you my views on your questions regarding ETL (extract, transform, load).

Data Lakes & Data Warehouse where is the difference?

Is ETL still practiced or did pre processing & cleansing replace it

What would replace ETL in Data Engineering?

How to become a data engineer? (check out my facebook note)

How to get experience training at home?

Real time analytics with RDBMS or HDFS?

What's the difference between Data Scientists & Data Analysts?

What to do to find internships or a full time job?

Data Scientist and Engineer in large and small companies where's the difference? Are Data Engineers generalists or specialists?

Just some questions I go over in this podcast.

You sent me over 100 Questions so, I finally worked up the guts to start with the Q&A videos. Answering your questions one by one.

Turns out it's a lot of fun :)

Without the proper tools and techniques of version control the team's efficiency goes down the drain. In this episode I talk about how tools like Jira enable you to collect bugs, future features or change requests. How they enable you to create and organize versions, add items to a version and assign items to developers. Once this is done, the team can efficiently start coding with the help of source code management systems like GitHub. How does all that work? Check out this episode to find out :)

You need to become comfortable with distributed processing. Data Science or the Internet of Things, the amount of data that is getting produced and processed grows like crazy. In this podcast I talk about how a platform for distributed processing looks like.

I talk about the different layers that need parallelization, as well as the tools you can use for on premise installations or clouds like AWS, Azure or Google Cloud. Big Data tools like Kafka, Spark or server less like Kinesis or Lambda functions.

For me, school and university was hard. The lectures, sitting down and getting told how things work.

Reading books and learning dry stuff was a drag. I was never good at writing tests.

Some people excel at this. I was often envious.

Over the years I found out what my problem is. I learn differently.

I am a learning by doing guy. What does that means and how am I dealing with it?

Check out this episode.

Maybe you have the same problem.

Becoming an expert in single skill is not the way to go for a data engineer. In this episode I talk about which talents go good together in terms of technical and personal ones. So, that you build up a stack of knowledge that will make you a great data engineer.

Strong APIs make a good platform. In this episode I talk about why you need APIs and why Twitter is a great example. Especially JSON APIs are my personal favorite. Because JSON is also important in the Big Data world, for instance in log analytics. How? Check out this episode!

Security is everything! That's why today, I took some time to give you some tips about how to make a good design. The Lambda Architecture with stream and batch processing is one of the cornerstones for Big Data and Data Science. How does that fit into a security zone design? Check out this episode :)

The understanding of how information is transported over the network is super important. OS wise you will mostly encounter Linux so here are some important Linux basics you need to know.

Firewalls, Ports, IP-Adresses, Routers and Switches, only a few things I talk about in this podcast. Networking infrastructure also matters for Big Data systems like Hadoop, Kafka and Spark.

Knowing the hardware is super important for a data engineer. Even if you are using cloud servers. CPU, RAM, GPU, HDD, SSD...

Especially the GPU is a great help to Data Scientists who are doing machine learning.