Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Advanced Search Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Mastering Hadoop
  • Toc
  • feedback
Mastering Hadoop

Mastering Hadoop

By : Karanth
4 (3)
close
Mastering Hadoop

Mastering Hadoop

4 (3)
By: Karanth

Overview of this book

Do you want to broaden your Hadoop skill set and take your knowledge to the next level? Do you wish to enhance your knowledge of Hadoop to solve challenging data processing problems? Are your Hadoop jobs, Pig scripts, or Hive queries not working as fast as you intend? Are you looking to understand the benefits of upgrading Hadoop? If the answer is yes to any of these, this book is for you. It assumes novice-level familiarity with Hadoop.
Table of Contents (15 chapters)
close
close
Preface
Preface
What this book covers
What you need for this book?
Who this book is for
Conventions
Reader feedback
Customer support
Free Chapter
1
1. Hadoop 2.X
1. Hadoop 2.X
The inception of Hadoop
The evolution of Hadoop
Hadoop 2.X
Hadoop distributions
Summary
2
2. Advanced MapReduce
2. Advanced MapReduce
MapReduce input
The RecordReader class
Hadoop's "small files" problem
Filtering inputs
The Map task
The Reduce task
MapReduce output
MapReduce job counters
Handling data joins
Summary
3
3. Advanced Pig
3. Advanced Pig
Pig versus SQL
Different modes of execution
Complex data types in Pig
Compiling Pig scripts
Development and debugging aids
The advanced Pig operators
User-defined functions
Pig performance optimizations
Best practices
Summary
4
4. Advanced Hive
4. Advanced Hive
The Hive architecture
Data types
File formats
The data model
Hive query optimizers
Advanced DML
UDF, UDAF, and UDTF
Summary
5
5. Serialization and Hadoop I/O
5. Serialization and Hadoop I/O
Data serialization in Hadoop
Avro serialization
File formats
Compression
Summary
6
6. YARN – Bringing Other Paradigms to Hadoop
chevron up
6. YARN – Bringing Other Paradigms to Hadoop
The YARN architecture
Developing YARN applications
Monitoring YARN
Job scheduling in YARN
YARN commands
Summary
7
7. Storm on YARN – Low Latency Processing in Hadoop
7. Storm on YARN – Low Latency Processing in Hadoop
Batch processing versus streaming
Apache Storm
Storm on YARN
Summary
8
8. Hadoop on the Cloud
8. Hadoop on the Cloud
Cloud computing characteristics
Hadoop on the cloud
Amazon Elastic MapReduce (EMR)
Summary
9
9. HDFS Replacements
9. HDFS Replacements
HDFS – advantages and drawbacks
Amazon AWS S3
Implementing a filesystem in Hadoop
Implementing an S3 native filesystem in Hadoop
Summary
10
10. HDFS Federation
10. HDFS Federation
Limitations of the older HDFS architecture
Architecture of HDFS Federation
HDFS high availability
HDFS block placement
Summary
11
11. Hadoop Security
11. Hadoop Security
The security pillars
Authentication in Hadoop
Authorization in Hadoop
Data confidentiality in Hadoop
Audit logging in Hadoop
Summary
12
12. Analytics Using Hadoop
12. Analytics Using Hadoop
Data analytics workflow
Machine learning
Apache Mahout
Document analysis using Hadoop and Mahout
RHadoop
Summary
13
A. Hadoop for Microsoft Windows
A. Hadoop for Microsoft Windows
Deploying Hadoop on Microsoft Windows
Summary
14
Index
Index

Job scheduling in YARN

Most cluster resources are multitenant in nature, that is, a number of teams or people share the cluster resources. Allocation of resources to satisfy the needs of all these tenants becomes important and is the responsibility of the scheduler. Individual clusters per team or person is not viable as they render poor utilization.

YARN provides a pluggable model to schedule policies. The initial versions of Hadoop had a simple First in First Out (FIFO) scheduler. However, FIFO was found to be inadequate in dealing with the complexities of multitenancy. We will discuss two other scheduling strategies that are used in Hadoop today, CapacityScheduler and FairScheduler.

CapacityScheduler

The concept behind CapacityScheduler is to guarantee a tenant-promised capacity on a shared cluster. If other tenants utilize less than the requested capacity, the scheduler allows the tenant to tap into these unused resources. The number one goal of CapacityScheduler is not to allow a single...

End of Section 4
Next Section
bookmark search playlist font-size

Change the font size

margin-width

Change margin width

day-mode

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Delete Bookmark

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete