Big Data and Hadoop Developer Online Certification Training 

Getting started with Big-Data and Hadoop Developer

Welcome

Course Introduction

Course Objectives

Course Overview

Course Overview(contd)

Value to Professionals

Lessons Covered

Thank You

Introduction to Big Data and Hadoop

Objectives

Data Explosion

Types of Data

Need for Big Data

Data - The Most Valuable Resource

Big Data and Its Sources

Three Characteristics of Big Data

Characteristics of Big Data Technology

Appeal of Big Data Technology

Leveraging Multiple Sources of Data

Traditional IT Analytics Approach

Big Data Technology - Platform for Discovery and Exploration

Big Data Technology - Capabilities

Big Data - Use Cases

Handling Limitations of Big Data

Introduction to Hadoop

History and Milestones of Hadoop

Organizations Using Hadoop

Quiz

Summary

Thank You

Getting Started with Hadoop

Objectives

VMware Player - Introduction

VMware Player - Hardware Requirements

Steps to Install VMware Player

Install VMware Player - Step

Install VMware Player - Step

Install VMware Player - Step

Install VMware Player - Step

Install VMware Player - Step

Install VMware Player - Step

Install VMware Player - Step

Install VMware Player - Step

Install VMware Player - Step

Steps to Create a VM in VMware Player

Create a VM in a VMware Player - Step

Create a VM in a VMware Player - Step

Create a VM in a VMware Player - Step

Create a VM in a VMware Player - Step

Create a VM in a VMware Player - Step

Create a VM in a VMware Player - Step

Open a VM in VMware Player - Step

Open a VM in VMware Player - Step

Oracle VirtualBox to Open a VM

Open a VM using Oracle VirtualBox - Step

Open a VM using Oracle VirtualBox - Step

Open a VM using Oracle VirtualBox - Step

Open a VM using Oracle VirtualBox - Step

Business Scenario

Demo

Demo Summary

Summary

Thank You

Hadoop Architecture

Objectives

Key Terms

Hadoop Cluster Using Commodity Hardware

Hadoop Configuration

Hadoop Core Services

Apache Hadoop Core Components

Hadoop Core Components - HDFS

Hadoop Core Components - MapReduce

Regular File System vs HDFS

HDFS - Characteristics

HDFS - Key Features

HDFS Architecture

HDFS - Operation Principle

HDFS

File System Namespace

NameNode Operation

Data Block Split

Benefits of Data Block Approach

HDFS - Block Replication Architecture

Replication Method

Data Replication Topology

Data Replication Representation

HDFS Access

Business Scenario

Demo

Demo Summary

Quiz

Summary

Thank You

Hadoop Deployment

Objectives

Ubuntu Server - Introduction

Installation of Ubuntu Server .

Business Scenario

Demo

Demo Summary

Hadoop Installation - Prerequisites

Hadoop Installation

Hadoop Installation - Step

Hadoop Installation - Step

Hadoop Installation - Step

Hadoop Installation - Step

Hadoop Installation - Step

Hadoop Installation - Step

Hadoop Installation - Step

Hadoop Installation - Step (contd)

Hadoop Installation - Step

Hadoop Installation - Step (contd)

Hadoop Installation - Step (contd)

Hadoop Installation - Step

Hadoop Installation - Step (contd)

Hadoop Installation - Step

Hadoop Installation - Step (contd)

Hadoop Installation - Step

Hadoop Installation - Step

Hadoop Installation - Step (contd)

Demo

Demo Summary

Hadoop Multi-Node Installation - Prerequisites

Steps for Hadoop Multi-Node Installation

Hadoop Multi-Node Installation - Steps and

Hadoop Multi-Node Installation - Step

Hadoop Multi-Node Installation - Step (contd)

Hadoop Multi-Node Installation - Step

Hadoop Multi-Node Installation - Step ( contd)

Hadoop Multi-Node Installation - Step (contd)

Single-Node Cluster vs Multi-Node Cluster

Demo

Demo Summary

Demo

Demo Summary

Demo

Demo Summary

Quiz

Summary

Thank You

Introduction to MapReduce

Objectives

MapReduce - Introduction

MapReduce - Analogy

MapReduce - Analogy (contd)

MapReduce - Example

Map Execution

Map Execution - Distributed Two Node Environment

MapReduce Essentials

MapReduce Jobs

MapReduce Engine

MapReduce and Associated Tasks

MapReduce Association with HDFS

Hadoop Job Work Interaction

Characteristics of MapReduce

Real-Time Uses of MapReduce

Prerequisites for Hadoop Installation in Ubuntu Desktop .

Steps to Install Hadoop

Business Scenario

Set up Environment for MapReduce Development

Small Data and Big Data

Uploading Small Data and Big Data

Demo

Demo Summary

Build MapReduce Program

Hadoop MapReduce Requirements

Hadoop MapReduce - Features

Hadoop MapReduce - Processes

Steps of Hadoop MapReduce

MapReduce - Responsibilities

MapReduce Java Programming in Eclipse

Create a New Project Step

Create a New Project Step

Create a New Project Step

Create a New Project Step

Create a New Project Step

Demo

Demo Summary

Demo

Demo Summary

Checking Hadoop Environment for MapReduce

Demo

Demo Summary

Demo

Demo Summary

Demo

Demo Summary

MapReduce v .

Quiz

Summary

Thank You

Advanced HDFS and MapReduce

Objectives

Advanced HDFS - Introduction

HDFS Benchmarking

HDFS Benchmarking (contd)

Setting Up HDFS Block Size

Setting Up HDFS Block Size - Step

Setting Up HDFS Block Size - Step

Decommissioning a DataNode

Decommissioning a DataNode - Step

Decommissioning a DataNode - Step

Decommissioning a DataNode - Step and

Business Scenario

Demo

Demo Summary

Advanced MapReduce

Interfaces

Data Types in Hadoop

InputFormats in MapReduce

OutputFormats in MapReduce

Distributed Cache

Using Distributed Cache - Step

Using Distributed Cache - Step

Using Distributed Cache - Step

Joins in MapReduce

Reduce Side Join

Reduce Side Join (contd)

Replicated Join

Replicated Join (contd)

Composite Join

Composite Join (contd)

Cartesian Product

Cartesian Product (contd)

Demo

Demo Summary

Quiz

Summary

Thank You

Pig

Objectives

Challenges Of MapReduce Development Using Java

Introduction To Pig

Components Of Pig

How Pig Works

Data Model

Data Model (contd)

Nested Data Model

Pig Execution Modes

Pig Interactive Modes

Salient Features

Pig vs SQL

Pig vs SQL - Example

Installing Pig Engine

Steps To Installing Pig Engine

Installing Pig Engine - Step

Installing Pig Engine - Step

Installing Pig Engine - Step

Installing Pig Engine - Step

Installing Pig Engine - Step

Run A Sample Program To Test Pig

Getting Datasets For Pig Development

Prerequisites To Set The Environment For Pig Latin

Prerequisites To Set The Environment For Pig Latin - Step

Prerequisites To Set The Environment For Pig Latin - Step

Prerequisites To Set The Environment For Pig Latin - Step

Loading And Storing Methods - Step

Loading and Storing Methods - Step

Script Interpretation

Filtering and Transforming

Grouping and Sorting

Combining and Splitting

Pig Commands

Business Scenario

Demo

Demo Summary

Demo

Demo Summary

Demo

Demo Summary

Demo

Demo Summary

Demo

Demo Summary

Demo

Demo Summary

Quiz

Summary

Thank You

Hive

Objectives

Need for Additional Data Warehousing System

Hive - Introduction

Hive - Characteristics

System Architecture and Components of Hive

Metastore

Metastore Configuration

Driver

Query Compiler

Query Optimizer

Execution Engine

Hive Server

Client Components

Basics of The Hive Query Language

Data Model - Tables

Data Model - External Tables

Data Types in Hive

Data Model - Partitions

Serialization and Deserialization

Hive File Formats

Hive Query Language - Select

Hive Query Language - JOIN and INSERT

Hive Installation - Step

Hive Installation - Step

Hive Installation - Step

Hive Installation - Step

Running Hive

Programming in Hive

Programming in Hive (contd)

Programming in Hive (contd)

Programming in Hive (contd)

Programming in Hive (contd)

Programming in Hive (contd)

Programming in Hive (contd)

Programming in Hive (contd)

Programming in Hive (contd)

Hive Query Language - Extensibility

User-Defined Function

Built-In Functions

Other Functions in Hive

MapReduce Scripts

UDF and UDAF vs MapReduce Scripts

Business Scenario

Demo

Demo Summary

Demo

Demo Summary

Demo

Demo Summary

Demo

Demo Summary

Quiz

Summary

Thank You

HBase

Objectives

HBase - Introduction

Characteristics of HBase

Companies Using HBase

HBase Architecture

HBase Architecture (contd)

Storage Model of HBase

Row Distribution of Data between RegionServers

Data Storage in HBase

Data Model

When to Use HBase

HBase vs RDBMS

Installation of HBase

Installation of HBase - Step

Installation of HBase - Steps and

Installation of HBase - Steps and

Installation of HBase - Steps and

Installation of HBase - Step

Configuration of HBase

Configuration of HBase - Step

Configuration of HBase - Step

Configuration of HBase - Steps and

Business Scenario

Demo

Demo Summary

Connecting to HBase

HBase Shell Commands

HBase Shell Commands (contd)

Quiz

Summary

Thank you

Commercial Distribution of Hadoop

Objectives

Cloudera - Introduction

Cloudera CDH

Downloading The Cloudera QuickStart Virtual Machine

Starting The Cloudera VM

Starting The Cloudera VM - Steps and

Starting The Cloudera VM - Steps and

Starting The Cloudera VM - Step

Starting The Cloudera VM - Step

Logging Into Hue

Logging Into Hue (contd)

Logging Into Hue (contd)

Cloudera Manager

Logging Into Cloudera Manager

Business Scenario

Demo

Demo Summary

Demo

Demo Summary

Hortonworks Data Platform

MapR Data Platform

Pivotal HD

IBM InfoSphere BigInsights

IBM InfoSphere BigInsights (contd)

Quiz

Summary

Thank You

ZooKeeper, Sqoop and Flume

Objectives

Introduction to ZooKeeper

Features of ZooKeeper

Challenges Faced in Distributed Applications

Coordination

Goals of ZooKeeper

Uses of ZooKeeper

ZooKeeper Entities

ZooKeeper Data Model

ZooKeeper Services

ZooKeeper Services (contd)

Client API Functions

Recipe : Cluster Management

Recipe : Leader Election

Recipe : Distributed Exclusive Lock

Business Scenario

Demo

Demo Summary

Why Sqoop

Why Sqoop (contd)

Benefits of Sqoop

Sqoop Processing

Sqoop Under The Hood

Importing Data Using Sqoop

Sqoop Import - Process

Sqoop Import - Process (contd)

Importing Data To Hive

Importing Data To HBase

Importing Data To HBase (contd)

Exporting Data From Hadoop Using Sqoop

Exporting Data From Hadoop Using Sqoop (contd)

Sqoop Connectors

Sample Sqoop Commands

Business Scenario

Demo

Demo Summary

Demo

Demo Summary

Why Flume

Apache Flume - Introduction

Flume Model

Flume - Goals

Scalability In Flume

Flume - Sample Use Cases

Business Scenario

Demo

Demo Summary

Quiz

Summary

Thank You

Ecosystem and Its Components

Objectives

Apache Hadoop Ecosystem

Apache Oozie

Apache Oozie Workflow

Apache Oozie Workflow (contd)

Introduction to Mahout

Why Mahout

Features of Mahout

Usage of Mahout

Usage of Mahout (contd)

Apache Cassandra

Why Apache Cassandra

Apache Spark

Apache Spark Tools

Key Concepts Related to Apache Spark

Apache Spark - Example

Hadoop Integration

Quiz

Summary

Thank You

Hadoop Administration, Troubleshooting and Security

Objectives

Typical Hadoop Core Cluster

Load Balancer

Commands Used in Hadoop Programming

Different Configuration Files of Hadoop Cluster

Properties of hadoop defaultxml

Different Configurations for Hadoop Cluster

Different Configurations for Hadoop Cluster (contd)

Port Numbers for Individual Hadoop Services

Performance Monitoring

Performance Tuning

Parameters of Performance Tuning

Troubleshooting and Log Observation

Apache Ambari

Key Features of Apache Ambari

Business Scenario

Demo

Demo Summary

Demo

Demo Summary

Hadoop Security - Kerberos

Kerberos - Authentication Mechanism

Kerberos Configuration

Data Confidentiality

Quiz

Summary

Thank You

Final Words

Final Words