Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Advanced Search Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Mastering Hadoop
  • Table Of Contents Toc
  • Feedback & Rating feedback
Mastering Hadoop

Mastering Hadoop

By : Karanth
4 (3)
Buy this Book
close
close
Mastering Hadoop

Mastering Hadoop

4 (3)
By: Karanth
Buy this Book

Overview of this book

Do you want to broaden your Hadoop skill set and take your knowledge to the next level? Do you wish to enhance your knowledge of Hadoop to solve challenging data processing problems? Are your Hadoop jobs, Pig scripts, or Hive queries not working as fast as you intend? Are you looking to understand the benefits of upgrading Hadoop? If the answer is yes to any of these, this book is for you. It assumes novice-level familiarity with Hadoop.
Table of Contents (15 chapters)
close
close
close
Preface
Icon
Preface
Icon
What this book covers
Icon
What you need for this book?
Icon
Who this book is for
Icon
Conventions
Icon
Reader feedback
Icon
Customer support
Free Chapter
1
1. Hadoop 2.X
Icon
1. Hadoop 2.X
Icon
The inception of Hadoop
Icon
The evolution of Hadoop
Icon
Hadoop 2.X
Icon
Hadoop distributions
Icon
Summary
2
2. Advanced MapReduce
Icon
2. Advanced MapReduce
Icon
MapReduce input
Icon
The RecordReader class
Icon
Hadoop's "small files" problem
Icon
Filtering inputs
Icon
The Map task
Icon
The Reduce task
Icon
MapReduce output
Icon
MapReduce job counters
Icon
Handling data joins
Icon
Summary
3
3. Advanced Pig
Icon
3. Advanced Pig
Icon
Pig versus SQL
Icon
Different modes of execution
Icon
Complex data types in Pig
Icon
Compiling Pig scripts
Icon
Development and debugging aids
Icon
The advanced Pig operators
Icon
User-defined functions
Icon
Pig performance optimizations
Icon
Best practices
Icon
Summary
4
4. Advanced Hive
Icon
4. Advanced Hive
Icon
The Hive architecture
Icon
Data types
Icon
File formats
Icon
The data model
Icon
Hive query optimizers
Icon
Advanced DML
Icon
UDF, UDAF, and UDTF
Icon
Summary
5
5. Serialization and Hadoop I/O
Icon
5. Serialization and Hadoop I/O
Icon
Data serialization in Hadoop
Icon
Avro serialization
Icon
File formats
Icon
Compression
Icon
Summary
6
6. YARN – Bringing Other Paradigms to Hadoop
Icon
6. YARN – Bringing Other Paradigms to Hadoop
Icon
The YARN architecture
Icon
Developing YARN applications
Icon
Monitoring YARN
Icon
Job scheduling in YARN
Icon
YARN commands
Icon
Summary
7
7. Storm on YARN – Low Latency Processing in Hadoop
Icon
7. Storm on YARN – Low Latency Processing in Hadoop
Icon
Batch processing versus streaming
Icon
Apache Storm
Icon
Storm on YARN
Icon
Summary
8
8. Hadoop on the Cloud
Icon
8. Hadoop on the Cloud
Icon
Cloud computing characteristics
Icon
Hadoop on the cloud
Icon
Amazon Elastic MapReduce (EMR)
Icon
Summary
9
9. HDFS Replacements
chevron up
Icon
9. HDFS Replacements
Icon
HDFS – advantages and drawbacks
Icon
Amazon AWS S3
Icon
Implementing a filesystem in Hadoop
Icon
Implementing an S3 native filesystem in Hadoop
Icon
Summary
10
10. HDFS Federation
Icon
10. HDFS Federation
Icon
Limitations of the older HDFS architecture
Icon
Architecture of HDFS Federation
Icon
HDFS high availability
Icon
HDFS block placement
Icon
Summary
11
11. Hadoop Security
Icon
11. Hadoop Security
Icon
The security pillars
Icon
Authentication in Hadoop
Icon
Authorization in Hadoop
Icon
Data confidentiality in Hadoop
Icon
Audit logging in Hadoop
Icon
Summary
12
12. Analytics Using Hadoop
Icon
12. Analytics Using Hadoop
Icon
Data analytics workflow
Icon
Machine learning
Icon
Apache Mahout
Icon
Document analysis using Hadoop and Mahout
Icon
RHadoop
Icon
Summary
13
A. Hadoop for Microsoft Windows
Icon
A. Hadoop for Microsoft Windows
Icon
Deploying Hadoop on Microsoft Windows
Icon
Summary
14
Index
Icon
Index

Implementing a filesystem in Hadoop

Based on the situation, it might be a necessity to replace HDFS with a filesystem of your choice. Hadoop provides out-of-the-box support for a few filesystems such as S3. HDFS replacement can be done either as a drop-in replacement or, as in the case with S3, seamless integration with the S3 file store for input and output.

In this section, we will re-implement the S3 native filesystem and extend Hadoop. The code in this section illustrates the steps on how HDFS replacement can be done. Error handling and other features related to S3 have been omitted for brevity.

The major steps in implementing a filesystem for Hadoop are as follows:

  1. The org.apache.hadoop.fs.FileSystem abstract class needs to be extended and all the abstract methods need to be overridden. There are out-of-the-box implementations for FilterFileSystem, NativeS3FileSystem, S3FileSystem, RawLocalFileSystem, FTPFileSystem, and ViewFileSystem.
  2. The open method returns an FsDataInputStream object...

Unlock full access

Continue reading for free

A Packt free trial gives you instant online access to our library of over 7000 practical eBooks and videos, constantly updated with the latest in tech
Start a 7-day FREE trial
End of Section 4
Next Section

Create a Note

Modal Close icon
You need to login to use this feature.
notes
bookmark search playlist font-size

Change the font size

margin-width

Change margin width

day-mode

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Delete Bookmark

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete

Delete Note

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete

Edit Note

Modal Close icon
Write a note (max 255 characters)
Cancel
Update Note

Confirmation

Modal Close icon
claim successful
Order Page Read Now

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY