During the workshop, you’ll act as a Big Data engineer working for a fictional company called StreamRockTM that creates a music streaming application (Spotify alike). The main goal of your work is to take advantage of Big Data technologies such as Hadoop, Spark, Hive, Kafka, Sqoop, Avro, Parquet to collect, store, clean and process various datasets about the users and the song they listened to. You’ll process our data to get data-driven answers to many business questions and power product features that StreamRockTM builds. Every exercise will be executed on a remote multi-node Hadoop cluster.
The workshop is highly focused on a practical experience. The instructor will also introduce you to his own practical experience gained while working with Big Data technologies for several years.
Software engineers who have (at least) basic knowledge of Python, Java and/or Scala and want to understand and develop distributed applications running on a Hadoop YARN cluster.
The training provides a carefully prepared mix of theory, exercises, demos, discussions, quizzes and&fun! We make sure that each participant is highly engaged in hands-on exercises, discussions and teamwork exercises.
Introduction to the Big Data and Apache Hadoop
Description of StreamRock company along with all its opportunities and challenges that come from the Big Data technologies.
Introduction to HDFS
Introduction to YARN
A short overview of MapReduce
Providing data-driven answers to business questions using SQL-like solution
Introduction to Apache Hive
Advanced aspects of Hive e.g. partitioning, bucketing, strict-mode, execution plan
Extending Hive with custom UDFs and SerDes
Hadoop File Formats (Avro, Parquet, ORC)
Implementing scalable ETL processes on the Hadoop cluster
Introduction to Apache Spark, Spark SQL, and Spark DataFrames
Bonus: Overview of Fast-SQL on Hadoop – solutions like Hive, Spark SQL, Impala, Presto and Tez
Introduction to Apache Sqoop
Real-time data collection with Apache Kafka
Introduction to Apache Oozie
* GetInData reserves the right to make any changes and adjustments to the presented agenda.
I do highly value substantive content of the course as well as great preparedness and layout. Knowledge passed in a ordered, consistent and effective way. Participants involvement during workshop sessions is the best indicator of this positive training!
Big Data workshops were led by real professionalists, tools and materials prepared in a way allowing participants to get down to the brass tacks straightaway without losing time. Attendees not disturbing each other and everyone can work comfortably and effectively. One can notice striking knowledge of the host and the fact that it comes from real professional work experience.
This is an excellent course and excellent teacher. Adam was well prepared, new the subject material, was good at transmitting his knowledge to us and had prepared exercises that added a lot of value to the sessions. I would rank this six if I could.
Professionally prepared and led courses. Coaches with vast experience in the presented realm.
Outstanding professor, the course was very well planned, he is very knowledgeable about what he taught. He talked about real-world cases and managed to get the whole class interested for 6 hours straight. Definitely one of the best courses that we have had in the masters.