Apache Spark 3 - Spark Programming in Python for Beginners

5.0 /5

(2 Ratings)

Offered byUDEMY

Apache Spark 3 - Spark Programming in Python for Beginners
at
UDEMY
Overview

Gain a comprehensive overview of the Apache Spark 3 principles and concepts

Duration	7 hours
Total fee	₹1,280
Mode of learning	Online
Difficulty level	Beginner
Credential	Certificate

Apache Spark 3 - Spark Programming in Python for Beginners
at
UDEMY
Highlights

Earn a Certificate of completion from Udemy
Learn from 14 downloadable resources and 2 articles
Get full lifetime access of the course material
Comes with 30 days money back guarantee and full lifetime access

Apache Spark 3 - Spark Programming in Python for Beginners
at
UDEMY
Course details

Skills you will learn

Spark Python Data Processing Software Testing Apache

Who should do this course?

For Software Engineers and Architects who are willing to design
For Programmers and developers who are aspiring to grow and learn Data Engineering using Apache Spark

What are the course deliverables?

Apache Spark Foundation and Spark Architecture
Working with Data Sources and Sinks
Using PyCharm IDE for Spark Development and Debugging
Data Engineering and Data Processing in Spark
Working with Data Frames and Spark SQL
Unit Testing, Managing Application Logs and Cluster Deployment

More about this course

Understand the Spark programming and apply that knowledge to build data engineering solutions
This course is example-driven and follows a working session like approach
We will be taking a live coding approach and explain all the needed concepts along the way

Apache Spark 3 - Spark Programming in Python for Beginners
at
UDEMY
Curriculum

Apache Spark Introduction

Understanding the Data Lake Landscape

What is Apache Spark - An Introduction and Overview

Installing and using apache spark

Mac Users - Apache Spark in Local Mode Command Line REPL

Windows Users - Apache Spark in Local Mode Command Line REPL

Mac Users - Apache Spark in the IDE - PyCharm

Windows Users - Apache Spark in the IDE - PyCharm

Apache Spark in Cloud - Databricks Community and Notebooks

Apache Spark in Anaconda - Jupyter Notebook

Spark Execution Model and Architecture

Execution Methods - How to Run Spark Programs?

Spark Distributed Processing Model - How your program runs?

Spark Execution Modes and Cluster Managers

Summarizing Spark Execution Models - When to use What?

Working with PySpark Shell - Demo

Installing Multi-Node Spark Cluster - Demo

Working with Notebooks in Cluster - Demo

Working with Spark Submit - Demo

Section Summary

Spark Programming Model and Developer Experience

Creating Spark Project Build Configuration

Configuring Spark Project Application Logs

Creating Spark Session

Configuring Spark Session

Data Frame Introduction

Data Frame Partitions and Executors

Spark Transformations and Actions

Spark Jobs Stages and Task

Spark Structured API Foundation

Introduction to Spark API

Introduction to Spark RDD API

Working with Spark SQL

Spark SQL Engine and Catalyst Optimizer

Section Summary

Spark Data Sources and Sinks

Spark Data Sources and Sinks

Spark DataFrameReader API

Reading CSV, JSON and Parquet files

Creating Spark DataFrame Schema

Spark DataFrameWriter API

Writing Your Data and Managing Layout

Spark Databases and Tables

Working with Spark SQL Tables

Spark Dataframe and Dataset Transformations

Introduction to Data Transformation

Working with Dataframe Rows

DataFrame Rows and Unit Testing

Dataframe Rows and Unstructured data

Working with Dataframe Columns

Creating and Using UDF

Misc Transformations

Aggregations in Apache Spark

Aggregating Dataframes

Grouping Aggregations

Windowing Aggregations

Spark Dataframe Joins

Dataframe Joins and column name ambiguity

Outer Joins in Dataframe

Internals of Spark Join and shuffle

Optimizing your joins

Implementing Bucket Joins

Apache Spark 3 - Spark Programming in Python for Beginners
at
UDEMY
Faculty details

Prashant Kumar Pandey

Prashant Kumar Pandey is passionate about helping people to learn and grow in their career by bridging the gap between their existing and required skills. In his quest to fulfill this mission, he is authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry.