Book Overview & Buying
Table Of Contents
Feedback & Rating

Building Big Data Pipelines with Apache Beam

By : Lukavský

3.7 (9)

Buy this Book

Building Big Data Pipelines with Apache Beam

3.7 (9)

By: Lukavský

Buy this Book

Overview of this book

Apache Beam is an open source unified programming model for implementing and executing data processing pipelines, including Extract, Transform, and Load (ETL), batch, and stream processing. This book will help you to confidently build data processing pipelines with Apache Beam. You’ll start with an overview of Apache Beam and understand how to use it to implement basic pipelines. You’ll also learn how to test and run the pipelines efficiently. As you progress, you’ll explore how to structure your code for reusability and also use various Domain Specific Languages (DSLs). Later chapters will show you how to use schemas and query your data using (streaming) SQL. Finally, you’ll understand advanced Apache Beam concepts, such as implementing your own I/O connectors. By the end of this book, you’ll have gained a deep understanding of the Apache Beam model and be able to apply it to solve problems.

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Download the color images

Conventions used

Get in touch

Share Your Thoughts

Section 1 Apache Beam: Essentials

Free Chapter

Chapter 1: Introduction to Data Processing with Apache Beam

Technical requirements

Why Apache Beam?

Writing your first pipeline

Running our pipeline against streaming data

Exploring the key properties of unbounded data

Measuring event time progress inside data streams

Assigning data to windows

Unifying batch and streaming data processing

Summary

Chapter 2: Implementing, Testing, and Deploying Basic Pipelines

Technical requirements

Setting up the environment for this book

Task 1 – Calculating the K most frequent words in a stream of lines of text

Task 2 – Calculating the maximal length of a word in a stream

Specifying the PCollection Coder object and the TypeDescriptor object

Understanding default triggers, on time, and closing behavior

Introducing the primitive PTransform object – Combine

Task 3 – Calculating the average length of words in a stream

Task 4 – Calculating the average length of words in a stream with fixed lookback

Ensuring pipeline upgradability

Task 5 – Calculating performance statistics for a sport activity tracking application

Introducing the primitive PTransform object – GroupByKey

Introducing the primitive PTransform object – Partition

Summary

Chapter 3: Implementing Pipelines Using Stateful Processing

Technical requirements

Task 6 – Using an external service for data augmentation

Introducing the primitive PTransform object – stateless ParDo

Task 7 – Batching queries to an external RPC service

Task 8 – Batching queries to an external RPC service with defined batch sizes

Introducing the primitive PTransform object – stateful ParDo

Using side outputs

Defining droppable data in Beam

Task 9 – Separating droppable data from the rest of the data processing

Task 10 – Separating droppable data from the rest of the data processing, part 2

Using side inputs

Summary

Section 2 Apache Beam: Toward Improving Usability

Chapter 4: Structuring Code for Reusability

Technical requirements

Explaining PTransform expansion

Task 11 – Enhancing SportTracker by runner motivation using side inputs

Introducing composite transform – CoGroupByKey

Task 12 – enhancing SportTracker by runner motivation using CoGroupByKey

Introducing the Join library DSL

Stream-to-stream joins explained

Task 13 – Writing a reusable PTransform – StreamingInnerJoin

Table-stream duality

Summary

Chapter 5: Using SQL for Pipeline Implementation

Technical requirements

Understanding schemas

Implementing our first streaming pipeline using SQL

Task 14 – Implementing SQLMaxWordLength

Task 15 – Implementing SchemaSportTracker

Task 16 – Implementing SQLSportTrackerMotivation

Further development of Apache Beam SQL

Summary

Chapter 6: Using Your Preferred Language with Portability

Technical requirements

Introducing the portability layer

Implementing our first pipelines in the Python SDK

Task 17 – Implementing MaxWordLength in the Python SDK

Python SDK type hints and coders

Task 18 – Implementing SportTracker in the Python SDK

Task 19 – Implementing RPCParDo in the Python SDK

Task 20 – Implementing SportTrackerMotivation in the Python SDK

Using the DataFrame API

Interactive programming using InteractiveRunner

Introducing and using cross-language pipelines

Summary

Section 3 Apache Beam: Advanced Concepts

Chapter 7: Extending Apache Beam's I/O Connectors

Technical requirements

Defining splittable DoFn as a unification for bounded and unbounded sources

Task 21 – Implementing our own splittable DoFn – a streaming file source

Task 22 – A non-I/O application of splittable DoFn – PiSampler

The legacy Source API and the Read transform

Writing a custom data sink

Summary

Chapter 8: Understanding How Runners Execute Pipelines

Describing the anatomy of an Apache Beam runner

Explaining the differences between classic and portable runners

Understanding how a runner handles state