Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Advanced Search Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Mastering Hadoop 3
  • Table Of Contents Toc
  • Feedback & Rating feedback
Mastering Hadoop 3

Mastering Hadoop 3

By : Wong, Singh, Kumar
5 (1)
Buy this Book
close
close
Mastering Hadoop 3

Mastering Hadoop 3

5 (1)
By: Wong, Singh, Kumar

Overview of this book

Apache Hadoop is one of the most popular big data solutions for distributed storage and for processing large chunks of data. With Hadoop 3, Apache promises to provide a high-performance, more fault-tolerant, and highly efficient big data processing platform, with a focus on improved scalability and increased efficiency. With this guide, you’ll understand advanced concepts of the Hadoop ecosystem tool. You’ll learn how Hadoop works internally, study advanced concepts of different ecosystem tools, discover solutions to real-world use cases, and understand how to secure your cluster. It will then walk you through HDFS, YARN, MapReduce, and Hadoop 3 concepts. You’ll be able to address common challenges like using Kafka efficiently, designing low latency, reliable message delivery Kafka systems, and handling high data volumes. As you advance, you’ll discover how to address major challenges when building an enterprise-grade messaging system, and how to use different stream processing systems along with Kafka to fulfil your enterprise goals. By the end of this book, you’ll have a complete understanding of how components in the Hadoop ecosystem are effectively integrated to implement a fast and reliable data pipeline, and you’ll be equipped to tackle a range of real-world problems in data pipelines.
Table of Contents (21 chapters)
close
close
close
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
Free Chapter
1
Section 1: Introduction to Hadoop 3
Section 1: Introduction to Hadoop 3
2
Journey to Hadoop 3
Journey to Hadoop 3
Hadoop origins and Timelines
Overview of Hadoop 3 and its features
Hadoop logical view
Hadoop distributions
Points to remember
Summary
3
Deep Dive into the Hadoop Distributed File System
Deep Dive into the Hadoop Distributed File System
Technical requirements
Defining HDFS
Deep dive into the HDFS architecture
NameNode internals
DataNode internals
Quorum Journal Manager (QJM)
HDFS high availability in Hadoop 3.x
Data management
HDFS reads and writes
Managing disk-skewed data in Hadoop 3.x
Lazy persist writes in HDFS
Erasure encoding in Hadoop 3.x
HDFS common interfaces
HDFS command reference
Points to remember
Summary
4
YARN Resource Management in Hadoop
YARN Resource Management in Hadoop
Architecture
Introduction to YARN job scheduling
FIFO scheduler
Capacity scheduler
Fair scheduler
Resource Manager high availability
Node labels
YARN Timeline server in Hadoop 3.x
Opportunistic containers in Hadoop 3.x
Docker containers in YARN
YARN REST APIs
YARN command reference
Summary
5
Internals of MapReduce
Internals of MapReduce
Technical requirements
Deep dive into the Hadoop MapReduce framework
YARN and MapReduce
MapReduce workflow in the Hadoop framework
Common MapReduce patterns
MapReduce use case
Optimizing MapReduce
Summary
6
Section 2: Hadoop Ecosystem
Section 2: Hadoop Ecosystem
7
SQL on Hadoop
SQL on Hadoop
Technical requirements
Presto – introduction
Hive
Impala
Summary 
8
Real-Time Processing Engines
Real-Time Processing Engines
Technical requirements
Spark
Apache Flink
Storm/Heron
Summary
9
Widely Used Hadoop Ecosystem Components
chevron up
Widely Used Hadoop Ecosystem Components
Technical requirements
Pig
HBase
Kafka
Flume
Summary
10
Section 3: Hadoop in the Real World
Section 3: Hadoop in the Real World
11
Designing Applications in Hadoop
Designing Applications in Hadoop
Technical requirements
File formats
Data compression
Serialization
Data ingestion
Data processing
Common batch processing pattern
Airflow for orchestration
Data governance
Summary
12
Real-Time Stream Processing in Hadoop
Real-Time Stream Processing in Hadoop
Technical requirements
What are streaming datasets?
Stream data ingestion
Common stream data processing patterns
Streaming design considerations
Micro-batch processing case study
Real-time processing case study
Summary
13
Machine Learning in Hadoop
Machine Learning in Hadoop
Technical requirements
Machine learning steps
Common machine learning challenges
Spark machine learning
Hadoop and R
Mahout
Machine learning case study in Spark
Summary
14
Hadoop in the Cloud
Hadoop in the Cloud
Technical requirements
Logical view of Hadoop in the cloud
Network
Managing resources
Data pipelines
High availability (HA)
Summary
15
Hadoop Cluster Profiling
Hadoop Cluster Profiling
Introduction to benchmarking and profiling
HDFS
NameNode
YARN
Hive
Mix-workloads
Summary
16
Section 4: Securing Hadoop
Section 4: Securing Hadoop
17
Who Can Do What in Hadoop
Who Can Do What in Hadoop
Hadoop security pillars
System security
Kerberos authentication
User authorization
List of security features that have been worked upon in Hadoop 3.0
Summary
18
Network and Data Security
Network and Data Security
Securing Hadoop networks
Encryption
Masking
Filtering
Summary
19
Monitoring Hadoop
Monitoring Hadoop
General monitoring
Security monitoring
Summary
20
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

Widely Used Hadoop Ecosystem Components

Since the invention of Hadoop, many tools have been developed around the Hadoop ecosystem. These tools are used for data ingestion, data processing, and storage, solving some of the problems Hadoop initially had. In this section, we will be focusing on Apache Pig, which is a distributed processing tool built on top of MapReduce. We will also look into two widely used ingestion tools, namely Apache Kafka and Apache Flume. We will discuss how they are used to bring data from multiple sources. Apache Hbase will be described in this chapter. We will cover the architecture details and how it fits into the CAP theorem. In this chapter, we will cover the following topics:

  • Apache Pig architecture 
  • Writing custom user-defined functions (UDF) in Pig
  • Apache HBase walkthrough 
  • CAP theorem
  • Apache Kafka internals 
  • Building producer...

Unlock full access

Continue reading for free

A Packt free trial gives you instant online access to our library of over 7000 practical eBooks and videos, constantly updated with the latest in tech
Start a 7-day FREE trial
End of Section 1
Next Section
bookmark search playlist download
font-size

Change the font size

margin-width

Change margin width

day-mode

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Delete Bookmark

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete

Confirmation

Modal Close icon
claim successful
Order Page Read Now

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY