Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Advanced Search Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Mastering Hadoop 3
  • Toc
  • feedback
Mastering Hadoop 3

Mastering Hadoop 3

By : Wong, Singh, Kumar
5 (1)
close
Mastering Hadoop 3

Mastering Hadoop 3

5 (1)
By: Wong, Singh, Kumar

Overview of this book

Apache Hadoop is one of the most popular big data solutions for distributed storage and for processing large chunks of data. With Hadoop 3, Apache promises to provide a high-performance, more fault-tolerant, and highly efficient big data processing platform, with a focus on improved scalability and increased efficiency. With this guide, you’ll understand advanced concepts of the Hadoop ecosystem tool. You’ll learn how Hadoop works internally, study advanced concepts of different ecosystem tools, discover solutions to real-world use cases, and understand how to secure your cluster. It will then walk you through HDFS, YARN, MapReduce, and Hadoop 3 concepts. You’ll be able to address common challenges like using Kafka efficiently, designing low latency, reliable message delivery Kafka systems, and handling high data volumes. As you advance, you’ll discover how to address major challenges when building an enterprise-grade messaging system, and how to use different stream processing systems along with Kafka to fulfil your enterprise goals. By the end of this book, you’ll have a complete understanding of how components in the Hadoop ecosystem are effectively integrated to implement a fast and reliable data pipeline, and you’ll be equipped to tackle a range of real-world problems in data pipelines.
Table of Contents (21 chapters)
close
close
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
Free Chapter
1
Section 1: Introduction to Hadoop 3
Section 1: Introduction to Hadoop 3
2
Journey to Hadoop 3
Journey to Hadoop 3
Hadoop origins and Timelines
Overview of Hadoop 3 and its features
Hadoop logical view
Hadoop distributions
Points to remember
Summary
3
Deep Dive into the Hadoop Distributed File System
Deep Dive into the Hadoop Distributed File System
Technical requirements
Defining HDFS
Deep dive into the HDFS architecture
NameNode internals
DataNode internals
Quorum Journal Manager (QJM)
HDFS high availability in Hadoop 3.x
Data management
HDFS reads and writes
Managing disk-skewed data in Hadoop 3.x
Lazy persist writes in HDFS
Erasure encoding in Hadoop 3.x
HDFS common interfaces
HDFS command reference
Points to remember
Summary
4
YARN Resource Management in Hadoop
YARN Resource Management in Hadoop
Architecture
Introduction to YARN job scheduling
FIFO scheduler
Capacity scheduler
Fair scheduler
Resource Manager high availability
Node labels
YARN Timeline server in Hadoop 3.x
Opportunistic containers in Hadoop 3.x
Docker containers in YARN
YARN REST APIs
YARN command reference
Summary
5
Internals of MapReduce
Internals of MapReduce
Technical requirements
Deep dive into the Hadoop MapReduce framework
YARN and MapReduce
MapReduce workflow in the Hadoop framework
Common MapReduce patterns
MapReduce use case
Optimizing MapReduce
Summary
6
Section 2: Hadoop Ecosystem
Section 2: Hadoop Ecosystem
7
SQL on Hadoop
SQL on Hadoop
Technical requirements
Presto – introduction
Hive
Impala
Summary 
8
Real-Time Processing Engines
Real-Time Processing Engines
Technical requirements
Spark
Apache Flink
Storm/Heron
Summary
9
Widely Used Hadoop Ecosystem Components
Widely Used Hadoop Ecosystem Components
Technical requirements
Pig
HBase
Kafka
Flume
Summary
10
Section 3: Hadoop in the Real World
Section 3: Hadoop in the Real World
11
Designing Applications in Hadoop
Designing Applications in Hadoop
Technical requirements
File formats
Data compression
Serialization
Data ingestion
Data processing
Common batch processing pattern
Airflow for orchestration
Data governance
Summary
12
Real-Time Stream Processing in Hadoop
Real-Time Stream Processing in Hadoop
Technical requirements
What are streaming datasets?
Stream data ingestion
Common stream data processing patterns
Streaming design considerations
Micro-batch processing case study
Real-time processing case study
Summary
13
Machine Learning in Hadoop
Machine Learning in Hadoop
Technical requirements
Machine learning steps
Common machine learning challenges
Spark machine learning
Hadoop and R
Mahout
Machine learning case study in Spark
Summary
14
Hadoop in the Cloud
Hadoop in the Cloud
Technical requirements
Logical view of Hadoop in the cloud
Network
Managing resources
Data pipelines
High availability (HA)
Summary
15
Hadoop Cluster Profiling
Hadoop Cluster Profiling
Introduction to benchmarking and profiling
HDFS
NameNode
YARN
Hive
Mix-workloads
Summary
16
Section 4: Securing Hadoop
Section 4: Securing Hadoop
17
Who Can Do What in Hadoop
Who Can Do What in Hadoop
Hadoop security pillars
System security
Kerberos authentication
User authorization
List of security features that have been worked upon in Hadoop 3.0
Summary
18
Network and Data Security
Network and Data Security
Securing Hadoop networks
Encryption
Masking
Filtering
Summary
19
Monitoring Hadoop
Monitoring Hadoop
General monitoring
Security monitoring
Summary
20
Other Books You May Enjoy
chevron up
Other Books You May Enjoy
Leave a review - let other readers know what you think

Other Books You May Enjoy

If you enjoyed this book, you may be interested in these other books by Packt:

Big Data Analytics with Hadoop 3
Sridhar Alla

ISBN: 978-1-78862-884-6

  • Explore the new features of Hadoop 3 along with HDFS, YARN, and MapReduce
  • Get well-versed with the analytical capabilities of Hadoop ecosystem using practical examples
  • Integrate Hadoop with R and Python for more efficient big data processing
  • Learn to use Hadoop with Apache Spark and Apache Flink for real-time data analytics
  • Set up a Hadoop cluster on AWS cloud
  • Perform big data analytics on AWS using Elastic Map Reduce

Apache Spark 2.x Machine Learning Cookbook
Siamak Amirghodsi et al.

ISBN: 978-1-78355-160-6

  • Get to know how Scala and Spark go hand-in-hand for developers when developing ML systems with Spark
  • Build a recommendation engine that scales with Spark
  • Find out how to build unsupervised clustering...
End of Section 1
Next Section
bookmark search playlist download
font-size

Change the font size

margin-width

Change margin width

day-mode

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Delete Bookmark

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete