Enterprise adoption of Hadoop is growing day by day. With increased adoption, there are a variety of application types that are using Hadoop for their enterprise goals. One such adoption is for applications that need to deal with data that amounts to only a few GBs. Keeping performance goals in mind with such small records would incur more latency costs when DISK I/O writes are involved during its execution—especially when such volumes of data can easily fit into memory without any DISK I/O. With the release of Hadoop 2.6, provisions for writes have been introduced that will use the off-heap memory of DataNodes. Eventually, data from memory will be flushed out to disk asynchronously. This will remove any expensive Disk I/O and computations for checksum while write operations are initiated from the HDFS client. Such asynchronous writes are called...
Mastering Hadoop 3
By :
Mastering Hadoop 3
By:
Overview of this book
Apache Hadoop is one of the most popular big data solutions for distributed storage and for processing large chunks of data. With Hadoop 3, Apache promises to provide a high-performance, more fault-tolerant, and highly efficient big data processing platform, with a focus on improved scalability and increased efficiency.
With this guide, you’ll understand advanced concepts of the Hadoop ecosystem tool. You’ll learn how Hadoop works internally, study advanced concepts of different ecosystem tools, discover solutions to real-world use cases, and understand how to secure your cluster. It will then walk you through HDFS, YARN, MapReduce, and Hadoop 3 concepts. You’ll be able to address common challenges like using Kafka efficiently, designing low latency, reliable message delivery Kafka systems, and handling high data volumes. As you advance, you’ll discover how to address major challenges when building an enterprise-grade messaging system, and how to use different stream processing systems along with Kafka to fulfil your enterprise goals.
By the end of this book, you’ll have a complete understanding of how components in the Hadoop ecosystem are effectively integrated to implement a fast and reliable data pipeline, and you’ll be equipped to tackle a range of real-world problems in data pipelines.
Table of Contents (21 chapters)
Preface
Journey to Hadoop 3
Deep Dive into the Hadoop Distributed File System
YARN Resource Management in Hadoop
Internals of MapReduce
Section 2: Hadoop Ecosystem
SQL on Hadoop
Real-Time Processing Engines
Widely Used Hadoop Ecosystem Components
Section 3: Hadoop in the Real World
Designing Applications in Hadoop
Real-Time Stream Processing in Hadoop
Machine Learning in Hadoop
Hadoop in the Cloud
Hadoop Cluster Profiling
Section 4: Securing Hadoop
Who Can Do What in Hadoop
Network and Data Security
Monitoring Hadoop
Other Books You May Enjoy
How would like to rate this book
Customer Reviews
