Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Advanced Search Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Mastering Hadoop
  • Toc
  • feedback
Mastering Hadoop

Mastering Hadoop

By : Karanth
4 (3)
close
Mastering Hadoop

Mastering Hadoop

4 (3)
By: Karanth

Overview of this book

Do you want to broaden your Hadoop skill set and take your knowledge to the next level? Do you wish to enhance your knowledge of Hadoop to solve challenging data processing problems? Are your Hadoop jobs, Pig scripts, or Hive queries not working as fast as you intend? Are you looking to understand the benefits of upgrading Hadoop? If the answer is yes to any of these, this book is for you. It assumes novice-level familiarity with Hadoop.
Table of Contents (15 chapters)
close
close
Preface
Preface
What this book covers
What you need for this book?
Who this book is for
Conventions
Reader feedback
Customer support
Free Chapter
1
1. Hadoop 2.X
1. Hadoop 2.X
The inception of Hadoop
The evolution of Hadoop
Hadoop 2.X
Hadoop distributions
Summary
2
2. Advanced MapReduce
2. Advanced MapReduce
MapReduce input
The RecordReader class
Hadoop's "small files" problem
Filtering inputs
The Map task
The Reduce task
MapReduce output
MapReduce job counters
Handling data joins
Summary
3
3. Advanced Pig
3. Advanced Pig
Pig versus SQL
Different modes of execution
Complex data types in Pig
Compiling Pig scripts
Development and debugging aids
The advanced Pig operators
User-defined functions
Pig performance optimizations
Best practices
Summary
4
4. Advanced Hive
4. Advanced Hive
The Hive architecture
Data types
File formats
The data model
Hive query optimizers
Advanced DML
UDF, UDAF, and UDTF
Summary
5
5. Serialization and Hadoop I/O
5. Serialization and Hadoop I/O
Data serialization in Hadoop
Avro serialization
File formats
Compression
Summary
6
6. YARN – Bringing Other Paradigms to Hadoop
6. YARN – Bringing Other Paradigms to Hadoop
The YARN architecture
Developing YARN applications
Monitoring YARN
Job scheduling in YARN
YARN commands
Summary
7
7. Storm on YARN – Low Latency Processing in Hadoop
7. Storm on YARN – Low Latency Processing in Hadoop
Batch processing versus streaming
Apache Storm
Storm on YARN
Summary
8
8. Hadoop on the Cloud
8. Hadoop on the Cloud
Cloud computing characteristics
Hadoop on the cloud
Amazon Elastic MapReduce (EMR)
Summary
9
9. HDFS Replacements
9. HDFS Replacements
HDFS – advantages and drawbacks
Amazon AWS S3
Implementing a filesystem in Hadoop
Implementing an S3 native filesystem in Hadoop
Summary
10
10. HDFS Federation
10. HDFS Federation
Limitations of the older HDFS architecture
Architecture of HDFS Federation
HDFS high availability
HDFS block placement
Summary
11
11. Hadoop Security
chevron up
11. Hadoop Security
The security pillars
Authentication in Hadoop
Authorization in Hadoop
Data confidentiality in Hadoop
Audit logging in Hadoop
Summary
12
12. Analytics Using Hadoop
12. Analytics Using Hadoop
Data analytics workflow
Machine learning
Apache Mahout
Document analysis using Hadoop and Mahout
RHadoop
Summary
13
A. Hadoop for Microsoft Windows
A. Hadoop for Microsoft Windows
Deploying Hadoop on Microsoft Windows
Summary
14
Index
Index

Summary

Security becomes a primary feature in a multitenant and distributed environment. Networks and resource sharing can potentially lead to information leaks via unauthorized access, malicious modifications, or even denial of service. Preempting attacks on Hadoop clusters can be done by enabling security features such as authentication, authorization, data protection, and data auditing.

The key takeaways from this chapter are as follows:

  • Post 0.20, Yahoo! introduced Hadoop security-related features for compliance, confidentiality, and fair usage in shared enterprise clusters.

  • Hadoop can now be configured for Kerberos-based authentication or simple authentication based on the topology and compliance requirements. User information can be retrieved from enterprise user stores such as LDAP or Active Directory.

  • Hadoop has both service-level and resource-level authorization built in. HDFS authorization is very similar to the UNIX-based file authorization model.

  • Hadoop provides data confidentiality...

End of Section 1
Next Section
bookmark search playlist font-size

Change the font size

margin-width

Change margin width

day-mode

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Delete Bookmark

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete