Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Advanced Search Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Hadoop Beginner's Guide
  • Table Of Contents Toc
  • Feedback & Rating feedback
Hadoop Beginner's Guide

Hadoop Beginner's Guide

3.7 (13)
Buy this Book
close
close
Hadoop Beginner's Guide

Hadoop Beginner's Guide

3.7 (13)
Buy this Book

Overview of this book

Data is arriving faster than you can process it and the overall volumes keep growing at a rate that keeps you awake at night. Hadoop can help you tame the data beast. Effective use of Hadoop however requires a mixture of programming, design, and system administration skills."Hadoop Beginner's Guide" removes the mystery from Hadoop, presenting Hadoop and related technologies with a focus on building working systems and getting the job done, using cloud services to do so when it makes sense. From basic concepts and initial setup through developing applications and keeping the system running as the data grows, the book gives the understanding needed to effectively use Hadoop to solve real world problems.Starting with the basics of installing and configuring Hadoop, the book explains how to develop applications, maintain the system, and how to use additional products to integrate with other systems.While learning different ways to develop applications to run on Hadoop the book also covers tools such as Hive, Sqoop, and Flume that show how Hadoop can be integrated with relational databases and log collection.In addition to examples on Hadoop clusters on Ubuntu uses of cloud services such as Amazon, EC2 and Elastic MapReduce are covered.
Table of Contents (19 chapters)
close
close
close
Hadoop Beginner's Guide
Icon
Hadoop Beginner's Guide
Credits
Icon
Credits
About the Author
Icon
About the Author
About the Reviewers
Icon
About the Reviewers
www.PacktPub.com
Icon
www.PacktPub.com
Preface
Icon
Preface
Free Chapter
1
What It's All About
Icon
What It's All About
Icon
Big data processing
Icon
Cloud computing with Amazon Web Services
Icon
Summary
2
Getting Hadoop Up and Running
Icon
Getting Hadoop Up and Running
Icon
Hadoop on a local Ubuntu host
Icon
Time for action – checking the prerequisites
Icon
Time for action – downloading Hadoop
Icon
Time for action – setting up SSH
Icon
Time for action – using Hadoop to calculate Pi
Icon
Time for action – configuring the pseudo-distributed mode
Icon
Time for action – changing the base HDFS directory
Icon
Time for action – formatting the NameNode
Icon
Time for action – starting Hadoop
Icon
Time for action – using HDFS
Icon
Time for action – WordCount, the Hello World of MapReduce
Icon
Using Elastic MapReduce
Icon
Time for action – WordCount on EMR using the management console
Icon
Comparison of local versus EMR Hadoop
Icon
Summary
3
Understanding MapReduce
chevron up
Icon
Understanding MapReduce
Icon
Key/value pairs
Icon
The Hadoop Java API for MapReduce
Icon
Writing MapReduce programs
Icon
Time for action – setting up the classpath
Icon
Time for action – implementing WordCount
Icon
Time for action – building a JAR file
Icon
Time for action – running WordCount on a local Hadoop cluster
Icon
Time for action – running WordCount on EMR
Icon
Time for action – WordCount the easy way
Icon
Walking through a run of WordCount
Icon
Time for action – WordCount with a combiner
Icon
Time for action – fixing WordCount to work with a combiner
Icon
Hadoop-specific data types
Icon
Time for action – using the Writable wrapper classes
Icon
Input/output
Icon
Summary
4
Developing MapReduce Programs
Icon
Developing MapReduce Programs
Icon
Using languages other than Java with Hadoop
Icon
Time for action – implementing WordCount using Streaming
Icon
Analyzing a large dataset
Icon
Time for action – summarizing the UFO data
Icon
Time for action – summarizing the shape data
Icon
Time for action – correlating of sighting duration to UFO shape
Icon
Time for action – performing the shape/time analysis from the command line
Icon
Time for action – using ChainMapper for field validation/analysis
Icon
Time for action – using the Distributed Cache to improve location output
Icon
Counters, status, and other output
Icon
Time for action – creating counters, task states, and writing log output
Icon
Summary
5
Advanced MapReduce Techniques
Icon
Advanced MapReduce Techniques
Icon
Simple, advanced, and in-between
Icon
Joins
Icon
Time for action – reduce-side join using MultipleInputs
Icon
Graph algorithms
Icon
Time for action – representing the graph
Icon
Time for action – creating the source code
Icon
Time for action – the first run
Icon
Time for action – the second run
Icon
Time for action – the third run
Icon
Time for action – the fourth and last run
Icon
Using language-independent data structures
Icon
Time for action – getting and installing Avro
Icon
Time for action – defining the schema
Icon
Time for action – creating the source Avro data with Ruby
Icon
Time for action – consuming the Avro data with Java
Icon
Time for action – generating shape summaries in MapReduce
Icon
Time for action – examining the output data with Ruby
Icon
Time for action – examining the output data with Java
Icon
Summary
6
When Things Break
Icon
When Things Break
Icon
Failure
Icon
Time for action – killing a DataNode process
Icon
Time for action – the replication factor in action
Icon
Time for action – intentionally causing missing blocks
Icon
Time for action – killing a TaskTracker process
Icon
Time for action – killing the JobTracker
Icon
Time for action – killing the NameNode process
Icon
Time for action – causing task failure
Icon
Time for action – handling dirty data by using skip mode
Icon
Summary
7
Keeping Things Running
Icon
Keeping Things Running
Icon
A note on EMR
Icon
Hadoop configuration properties
Icon
Time for action – browsing default properties
Icon
Setting up a cluster
Icon
Time for action – examining the default rack configuration
Icon
Time for action – adding a rack awareness script
Icon
Cluster access control
Icon
Time for action – demonstrating the default security
Icon
Managing the NameNode
Icon
Time for action – adding an additional fsimage location
Icon
Time for action – swapping to a new NameNode host
Icon
Managing HDFS
Icon
MapReduce management
Icon
Time for action – changing job priorities and killing a job
Icon
Scaling
Icon
Summary
8
A Relational View on Data with Hive
Icon
A Relational View on Data with Hive
Icon
Overview of Hive
Icon
Setting up Hive
Icon
Time for action – installing Hive
Icon
Using Hive
Icon
Time for action – creating a table for the UFO data
Icon
Time for action – inserting the UFO data
Icon
Time for action – validating the table
Icon
Time for action – redefining the table with the correct column separator
Icon
Time for action – creating a table from an existing file
Icon
Time for action – performing a join
Icon
Time for action – using views
Icon
Time for action – exporting query output
Icon
Time for action – making a partitioned UFO sighting table
Icon
Time for action – adding a new User Defined Function (UDF)
Icon
Hive on Amazon Web Services
Icon
Time for action – running UFO analysis on EMR
Icon
Summary
9
Working with Relational Databases
Icon
Working with Relational Databases
Icon
Common data paths
Icon
Setting up MySQL
Icon
Time for action – installing and setting up MySQL
Icon
Time for action – configuring MySQL to allow remote connections
Icon
Time for action – setting up the employee database
Icon
Getting data into Hadoop
Icon
Time for action – downloading and configuring Sqoop
Icon
Time for action – exporting data from MySQL to HDFS
Icon
Time for action – exporting data from MySQL into Hive
Icon
Time for action – a more selective import
Icon
Time for action – using a type mapping
Icon
Time for action – importing data from a raw query
Icon
Getting data out of Hadoop
Icon
Time for action – importing data from Hadoop into MySQL
Icon
Time for action – importing Hive data into MySQL
Icon
Time for action – fixing the mapping and re-running the export
Icon
AWS considerations
Icon
Summary
10
Data Collection with Flume
Icon
Data Collection with Flume
Icon
A note about AWS
Icon
Data data everywhere...
Icon
Time for action – getting web server data into Hadoop
Icon
Introducing Apache Flume
Icon
Time for action – installing and configuring Flume
Icon
Time for action – capturing network traffic in a log file
Icon
Time for action – logging to the console
Icon
Time for action – capturing the output of a command to a flat file
Icon
Time for action – capturing a remote file in a local flat file
Icon
Time for action – writing network traffic onto HDFS
Icon
Time for action – adding timestamps
Icon
Time for action – multi level Flume networks
Icon
Time for action – writing to multiple sinks
Icon
The bigger picture
Icon
Summary
11
Where to Go Next
Icon
Where to Go Next
Icon
What we did and didn't cover in this book
Icon
Upcoming Hadoop changes
Icon
Alternative distributions
Icon
Other Apache projects
Icon
Other programming abstractions
Icon
AWS resources
Icon
Sources of information
Icon
Summary
Pop Quiz Answers
Icon
Pop Quiz Answers
Icon
Chapter 3, Understanding MapReduce
Icon
Chapter 7, Keeping Things Running
Index
Icon
Index

Time for action – WordCount with a combiner

Let's add a combiner to our first WordCount example. In fact, let's use our reducer as the combiner. Since the combiner must have the same interface as the reducer, this is something you'll often see, though note that the type of processing involved in the reducer will determine if it is a true candidate for a combiner; we'll discuss this later. Since we are looking to count word occurrences, we can do a partial count on the map node and pass these subtotals to the reducer.

  1. Copy WordCount1.java to WordCount2.java and change the driver class to add the following line between the definition of the Mapper and Reducer classes:

            job.setCombinerClass(WordCountReducer.class);

  2. Also change the class name to WordCount2 and then compile it.

    $ javac WordCount2.java

  3. Create the JAR file.

    $ jar cvf wc2.jar WordCount2*class

  4. Run the job on Hadoop.

    $ hadoop jar wc2.jar WordCount2 test.txt output

  5. Examine the output.

    $ hadoop fs -cat output/part-r-00000

What just happened...

Unlock full access

Continue reading for free

A Packt free trial gives you instant online access to our library of over 7000 practical eBooks and videos, constantly updated with the latest in tech
Start a 7-day FREE trial
End of Section 12
Next Section

Create a Note

Modal Close icon
You need to login to use this feature.
notes
bookmark search playlist font-size

Change the font size

margin-width

Change margin width

day-mode

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Delete Bookmark

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete

Delete Note

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete

Edit Note

Modal Close icon
Write a note (max 255 characters)
Cancel
Update Note

Confirmation

Modal Close icon
claim successful
Order Page Read Now

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY