Data Engineering Essentials Hands-on - SQL, Python and Spark 

You will start with setting up self-support Data Engineering Labs either on GCP or Cloud9 so that you can learn the key skills related to Data Engineering with a lot of practice leveraging tasks and exercises provided by us. As you pass the sections related to SQL and Python, you will also be guided to set up Hadoop and Spark Lab.

Provision GCP Server or AWS Cloud9 Instance

Setup Docker to host Postgres Database

Setup Postgres Database to practice SQL

Setup Jupyter Lab

Once Jupyter Lab is setup, you can upload the Jupyter Notebooks and start practicing all the key skills related to Data Engineering.

It is important for one to be proficient with SQL to take care of building data engineering pipelines. SQL is used for understanding the data, perform ad-hoc analysis, and also in building data engineering pipelines.

Getting Started with Postgres

Basic Database Operations (CRUD or Insert, Update, Delete)

Writing Basic SQL Queries (Filtering, Joins, and Aggregations)

Creating Tables and Indexes

Partitioning Tables and Indexes

Predefined Functions (String Manipulation, Date Manipulation, and other functions)

Writing Advanced SQL Queries

Python is the most preferred programming language to develop data engineering applications. As part of several sections related to Python, you will be learning most of the important aspects of Python to build data engineering applications effectively.

Perform Database Operations

Getting Started with Python

Basic Programming Constructs

Predefined Functions

Overview of Collections - list and set

Overview of Collections - dict and tuple

Manipulating Collections using loops

Understanding Map Reduce Libraries

Overview of Pandas Libraries

Database Programming - CRUD Operations

Database Programming - Batch Operations

The most common approach to build data engineering applications at scale is by using Spark integrated with HDFS and YARN. Before getting into data engineering using Spark and Hadoop, we need to set up an environment to practice data engineering using Spark. As part of this section, we will primarily focus on setting up a single node cluster to learn key skills related to data engineering using distributed frameworks such as Spark and Hadoop.

Setup Single Node Hadoop Cluster

Setup Hive and Spark on Single Node Cluster

As part of this section, you will primarily focus on HDFS commands so that we can copy files into HDFS. The data copied into HDFS will be used as part of building data engineering pipelines using Spark and Hadoop with Python as Programming Language.

Overview of HDFS Commands

Let us, deep-dive into Spark SQL to understand how it can be used to build Data Engineering Pipelines. Spark with SQL will provide us the ability to leverage distributed computing capabilities of Spark coupled with easy-to-use developer-friendly SQL-style syntax.

Basic Transformations

Managing Tables - Basic DDL and DML

Managing Tables - DML and Partitioning

Overview of Spark SQL Functions

Windowing Functions

Spark Data Frame APIs are an alternative way of building Data Engineering applications at scale leveraging distributed computing capabilities of Spark. Data Engineers from application development backgrounds might prefer Data Frame APIs over Spark SQL to build Data Engineering applications.

Data Processing Overview

Processing Column Data

Joining Data Sets

Windowing Functions - Aggregations, Ranking, and Analytic Functions

Spark Metastore Databases and Tables

Desired Audience for this Data Engineering Essentials course

People from different backgrounds can aim to become Data Engineers. We cover most of the Data Engineering essentials for the aspirants who want to get into the IT field as Data Engineers as well as professionals who want to propel their career towards Data Engineering from legacy technologies.

College students and entry-level professionals to get hands-on expertise with respect to Data Engineering. This course will provide enough skills to face interviews for entry-level data engineers.

Experienced application developers to gain expertise related to Data Engineering.

Conventional Data Warehouse Developers, ETL Developers, Database Developers, PL/SQL Developers to gain enough skills to transition to be successful Data Engineers.

Testers to improve their testing capabilities related to Data Engineering applications.

Any other hands-on IT Professional who wants to get knowledge about Data Engineering with Hands-On Practice.