Book Overview & Buying
Table Of Contents
Feedback & Rating

Learning Apache Apex

By : Gundabattula, Thomas Weise, Munagala V. Ramanath, David Yan, Kenneth Knowles

5 (1)

Buy this Book

Learning Apache Apex

5 (1)

By: Gundabattula, Thomas Weise, Munagala V. Ramanath, David Yan, Kenneth Knowles

Buy this Book

Overview of this book

Apache Apex is a next-generation stream processing framework designed to operate on data at large scale, with minimum latency, maximum reliability, and strict correctness guarantees. Half of the book consists of Apex applications, showing you key aspects of data processing pipelines such as connectors for sources and sinks, and common data transformations. The other half of the book is evenly split into explaining the Apex framework, and tuning, testing, and scaling Apex applications. Much of our economic world depends on growing streams of data, such as social media feeds, financial records, data from mobile devices, sensors and machines (the Internet of Things - IoT). The projects in the book show how to process such streams to gain valuable, timely, and actionable insights. Traditional use cases, such as ETL, that currently consume a significant chunk of data engineering resources are also covered. The final chapter shows you future possibilities emerging in the streaming space, and how Apache Apex can contribute to it.

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

Introduction to Apex

Unbounded data and continuous processing

Use cases and case studies

Application Model and API

Value proposition of Apex

Summary

Getting Started with Application Development

Development process and methodology

Setting up the development environment

Creating a new Maven project

Application specifications

Custom operator development

Application configuration

Testing in the IDE

Running the application on YARN

Working on the cluster

Summary

The Apex Library

An overview of the library

Integrations

Transformations

Summary

Scalability, Low Latency, and Performance

Partitioning and how it works

Elasticity

Partitioning toolkit

Custom dynamic partitioning

Performance optimizations

Low-latency versus throughput

Sample application for dynamic partitioning

Performance – other aspects for custom operators

Summary

Fault Tolerance and Reliability

Distributed systems need to be resilient

Fault-tolerance components and mechanism in Apex

Checkpointing

Processing guarantees

Summary

Example Project – Real-Time Aggregation and Visualization

Streaming ETL and beyond

The application pattern in a real-world use case

Analyzing Twitter feed

Running the application

The Pub/Sub server

Grafana visualization

Summary

Example Project – Real-Time Ride Service Data Processing

The goal

Datasource

The pipeline

Simulation of a real-time feed using historical data

Parsing the data

Looking up of the zip code and preparing for the windowing operation

Windowed operator configuration

Serving the data with WebSocket

Running the application

Running the application on GCP Dataproc

Summary

Example Project – ETL Using SQL

The application pipeline

Building and running the application

Application configuration

The application code

Partitioning

Application testing

Understanding application logs

Calcite integration

Summary

Introduction to Apache Beam

Beam concepts

WordCount in Apache Beam

Running Apache Beam WordCount on Apache Apex

Summary

The Future of Stream Processing

Lower barrier for building streaming pipelines

Summary

Customer Reviews

5 (1)

5 star

100%

4 star

3 star

2 star

1 star

Creating a new Maven project

Apex applications are packaged in a special ZIP file format that contains everything needed for an application to be launched on a cluster (dependency jars, configuration files, and so on). It is roughly comparable to the uber jar approach that some other frameworks employ, with the difference that dependencies in the Apex package remain as individual JAR files, rather than being flattened into a standard JAR.

More information about Apex application packages can be found at http://apex.apache.org/docs/apex/application_packages/#apache-apex-packages.

It would be a rather involved task to set up a new Maven project from scratch. The Apex application archetype simplifies the process of creating an application skeleton for the expected artifact structure. Here is an example of the Maven command to create an Apex application archetype: