Book Overview & Buying
Table Of Contents
Feedback & Rating

Learning Apache Apex

By : Gundabattula, Thomas Weise, Munagala V. Ramanath, David Yan, Kenneth Knowles

5 (1)

Buy this Book

Learning Apache Apex

5 (1)

By: Gundabattula, Thomas Weise, Munagala V. Ramanath, David Yan, Kenneth Knowles

Buy this Book

Overview of this book

Apache Apex is a next-generation stream processing framework designed to operate on data at large scale, with minimum latency, maximum reliability, and strict correctness guarantees. Half of the book consists of Apex applications, showing you key aspects of data processing pipelines such as connectors for sources and sinks, and common data transformations. The other half of the book is evenly split into explaining the Apex framework, and tuning, testing, and scaling Apex applications. Much of our economic world depends on growing streams of data, such as social media feeds, financial records, data from mobile devices, sensors and machines (the Internet of Things - IoT). The projects in the book show how to process such streams to gain valuable, timely, and actionable insights. Traditional use cases, such as ETL, that currently consume a significant chunk of data engineering resources are also covered. The final chapter shows you future possibilities emerging in the streaming space, and how Apache Apex can contribute to it.

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

Introduction to Apex

Unbounded data and continuous processing

Use cases and case studies

Application Model and API

Value proposition of Apex

Summary

Getting Started with Application Development

Development process and methodology

Setting up the development environment

Creating a new Maven project

Application specifications

Custom operator development

Application configuration

Testing in the IDE

Running the application on YARN

Working on the cluster

Summary

The Apex Library

An overview of the library

Integrations

Transformations

Summary

Scalability, Low Latency, and Performance

Partitioning and how it works

Elasticity

Partitioning toolkit

Custom dynamic partitioning

Performance optimizations

Low-latency versus throughput

Sample application for dynamic partitioning

Performance – other aspects for custom operators

Summary

Fault Tolerance and Reliability

Distributed systems need to be resilient

Fault-tolerance components and mechanism in Apex

Checkpointing

Processing guarantees

Summary

Example Project – Real-Time Aggregation and Visualization

Streaming ETL and beyond

The application pattern in a real-world use case

Analyzing Twitter feed

Running the application

The Pub/Sub server

Grafana visualization

Summary

Example Project – Real-Time Ride Service Data Processing

The goal

Datasource

The pipeline

Simulation of a real-time feed using historical data

Parsing the data

Looking up of the zip code and preparing for the windowing operation

Windowed operator configuration

Serving the data with WebSocket

Running the application

Running the application on GCP Dataproc

Summary

Example Project – ETL Using SQL

The application pipeline

Building and running the application

Application configuration

The application code

Partitioning

Application testing

Understanding application logs

Calcite integration

Summary

Introduction to Apache Beam

Beam concepts

WordCount in Apache Beam

Running Apache Beam WordCount on Apache Apex

Summary

The Future of Stream Processing

Lower barrier for building streaming pipelines

Summary

Customer Reviews

5 (1)

5 star

100%

4 star

3 star

2 star

1 star

Fault-tolerance components and mechanism in Apex

In Chapter 2, Getting Started with Application Development, we looked at the deployment of an Apex application when it is executing on a YARN cluster. Let's revisit the diagram to see which type of failures may occur and how they are handled by the system:

The client is only required for launching the application; it is not involved in the execution of the DAG on the cluster, and failure of the client node does not affect the pipeline. Since Apex is running on YARN, let's first see how YARN supports resilient applications (from a user's perspective).

YARN consists of a resource manager (RM) and node managers (NM). Each YARN cluster node has a node manager service running, which communicates with the resource manager.