Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Advanced Search Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Learning Hadoop 2
  • Toc
  • feedback
Learning Hadoop 2

Learning Hadoop 2

By : GABRIELE MODENA
3.8 (4)
close
Learning Hadoop 2

Learning Hadoop 2

3.8 (4)
By: GABRIELE MODENA

Overview of this book

If you are a system or application developer interested in learning how to solve practical problems using the Hadoop framework, then this book is ideal for you. You are expected to be familiar with the Unix/Linux command-line interface and have some experience with the Java programming language. Familiarity with Hadoop would be a plus.
Table of Contents (13 chapters)
close
close
Preface
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Free Chapter
1
1. Introduction
1. Introduction
A note on versioning
The background of Hadoop
Components of Hadoop
Hadoop 2 – what's the big deal?
Distributions of Apache Hadoop
A dual approach
AWS – infrastructure on demand from Amazon
Getting started
Running the examples
Data processing with Hadoop
Summary
2
2. Storage
2. Storage
The inner workings of HDFS
Command-line access to the HDFS filesystem
Protecting the filesystem metadata
Apache ZooKeeper – a different type of filesystem
Automatic NameNode failover
HDFS snapshots
Hadoop filesystems
Managing and serializing data
Storing data
Summary
3
3. Processing – MapReduce and Beyond
3. Processing – MapReduce and Beyond
MapReduce
Java API to MapReduce
Writing MapReduce programs
Walking through a run of a MapReduce job
YARN
YARN in the real world – Computation beyond MapReduce
Summary
4
4. Real-time Computation with Samza
4. Real-time Computation with Samza
Stream processing with Samza
Summary
5
5. Iterative Computation with Spark
5. Iterative Computation with Spark
Apache Spark
The Spark ecosystem
Processing data with Apache Spark
Comparing Samza and Spark Streaming
Summary
6
6. Data Analysis with Apache Pig
6. Data Analysis with Apache Pig
An overview of Pig
Getting started
Running Pig
Fundamentals of Apache Pig
Programming Pig
Extending Pig (UDFs)
Analyzing the Twitter stream
Summary
7
7. Hadoop and SQL
chevron up
7. Hadoop and SQL
Why SQL on Hadoop
Prerequisites
Hive architecture
Hive and Amazon Web Services
Extending HiveQL
Programmatic interfaces
Stinger initiative
Impala
Summary
8
8. Data Lifecycle Management
8. Data Lifecycle Management
What data lifecycle management is
Building a tweet analysis capability
Challenges of external data
Collecting additional data
Pulling it all together
Summary
9
9. Making Development Easier
9. Making Development Easier
Choosing a framework
Hadoop streaming
Kite Data
Apache Crunch
Summary
10
10. Running a Hadoop Cluster
10. Running a Hadoop Cluster
I'm a developer – I don't care about operations!
Cloudera Manager
Ambari – the open source alternative
Operations in the Hadoop 2 world
Sharing resources
Building a physical cluster
Building a cluster on EMR
Cluster tuning
Security
Monitoring
Troubleshooting
Summary
11
11. Where to Go Next
11. Where to Go Next
Alternative distributions
Other computational frameworks
Other interesting projects
Other programming abstractions
AWS resources
Sources of information
Summary
12
Index
Index

Prerequisites

Before diving into specific technologies, let's generate some data that we'll use in the examples throughout this chapter. We'll create a modified version of a former Pig script as the main functionality for this. The script in this chapter assumes that the Elephant Bird JARs used previously are available in the /jar directory on HDFS. The full source code is at https://github.com/learninghadoop2/book-examples/blob/master/ch7/extract_for_hive.pig, but the core of extract_for_hive.pig is as follows:

-- load JSON data
tweets = load '$inputDir' using  com.twitter.elephantbird.pig.load.JsonLoader('-nestedLoad');
-- Tweets
tweets_tsv = foreach tweets {
generate 
    (chararray)CustomFormatToISO($0#'created_at', 
'EEE MMMM d HH:mm:ss Z y') as dt, 
    (chararray)$0#'id_str', 
(chararray)$0#'text' as text, 
    (chararray)$0#'in_reply_to', 
(boolean)$0#'retweeted' as is_retweeted, 
(chararray)$0#'user'#'id_str' as user_id, (chararray)$0#'place'#'id' as place_id;
}
store tweets_tsv...

Unlock full access

Continue reading for free

A Packt free trial gives you instant online access to our library of over 7000 practical eBooks and videos, constantly updated with the latest in tech
Start a 7-day FREE trial
End of Section 3
Next Section
bookmark search playlist download
font-size

Change the font size

margin-width

Change margin width

day-mode

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Delete Bookmark

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete