Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Advanced Search Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Building Big Data Pipelines with Apache Beam
  • Toc
  • feedback
Building Big Data Pipelines with Apache Beam

Building Big Data Pipelines with Apache Beam

By : Lukavský
3.7 (9)
close
Building Big Data Pipelines with Apache Beam

Building Big Data Pipelines with Apache Beam

3.7 (9)
By: Lukavský

Overview of this book

Apache Beam is an open source unified programming model for implementing and executing data processing pipelines, including Extract, Transform, and Load (ETL), batch, and stream processing. This book will help you to confidently build data processing pipelines with Apache Beam. You’ll start with an overview of Apache Beam and understand how to use it to implement basic pipelines. You’ll also learn how to test and run the pipelines efficiently. As you progress, you’ll explore how to structure your code for reusability and also use various Domain Specific Languages (DSLs). Later chapters will show you how to use schemas and query your data using (streaming) SQL. Finally, you’ll understand advanced Apache Beam concepts, such as implementing your own I/O connectors. By the end of this book, you’ll have gained a deep understanding of the Apache Beam model and be able to apply it to solve problems.
Table of Contents (13 chapters)
close
close
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Download the color images
Conventions used
Get in touch
Share Your Thoughts
1
Section 1 Apache Beam: Essentials
Section 1 Apache Beam: Essentials
Free Chapter
2
Chapter 1: Introduction to Data Processing with Apache Beam
Chapter 1: Introduction to Data Processing with Apache Beam
Technical requirements
Why Apache Beam?
Writing your first pipeline
Running our pipeline against streaming data
Exploring the key properties of unbounded data
Measuring event time progress inside data streams
Assigning data to windows
Unifying batch and streaming data processing
Summary
3
Chapter 2: Implementing, Testing, and Deploying Basic Pipelines
Chapter 2: Implementing, Testing, and Deploying Basic Pipelines
Technical requirements
Setting up the environment for this book
Task 1 – Calculating the K most frequent words in a stream of lines of text
Task 2 – Calculating the maximal length of a word in a stream
Specifying the PCollection Coder object and the TypeDescriptor object
Understanding default triggers, on time, and closing behavior
Introducing the primitive PTransform object – Combine
Task 3 – Calculating the average length of words in a stream
Task 4 – Calculating the average length of words in a stream with fixed lookback
Ensuring pipeline upgradability
Task 5 – Calculating performance statistics for a sport activity tracking application
Introducing the primitive PTransform object – GroupByKey
Introducing the primitive PTransform object – Partition
Summary
4
Chapter 3: Implementing Pipelines Using Stateful Processing
Chapter 3: Implementing Pipelines Using Stateful Processing
Technical requirements
Task 6 – Using an external service for data augmentation
Introducing the primitive PTransform object – stateless ParDo
Task 7 – Batching queries to an external RPC service
Task 8 – Batching queries to an external RPC service with defined batch sizes
Introducing the primitive PTransform object – stateful ParDo
Using side outputs
Defining droppable data in Beam
Task 9 – Separating droppable data from the rest of the data processing
Task 10 – Separating droppable data from the rest of the data processing, part 2
Using side inputs
Summary
5
Section 2 Apache Beam: Toward Improving Usability
Section 2 Apache Beam: Toward Improving Usability
6
Chapter 4: Structuring Code for Reusability
Chapter 4: Structuring Code for Reusability
Technical requirements
Explaining PTransform expansion
Task 11 – Enhancing SportTracker by runner motivation using side inputs
Introducing composite transform – CoGroupByKey
Task 12 – enhancing SportTracker by runner motivation using CoGroupByKey
Introducing the Join library DSL
Stream-to-stream joins explained
Task 13 – Writing a reusable PTransform – StreamingInnerJoin
Table-stream duality
Summary
7
Chapter 5: Using SQL for Pipeline Implementation
Chapter 5: Using SQL for Pipeline Implementation
Technical requirements
Understanding schemas
Implementing our first streaming pipeline using SQL
Task 14 – Implementing SQLMaxWordLength
Task 15 – Implementing SchemaSportTracker
Task 16 – Implementing SQLSportTrackerMotivation
Further development of Apache Beam SQL
Summary
8
Chapter 6: Using Your Preferred Language with Portability
Chapter 6: Using Your Preferred Language with Portability
Technical requirements
Introducing the portability layer
Implementing our first pipelines in the Python SDK
Task 17 – Implementing MaxWordLength in the Python SDK
Python SDK type hints and coders
Task 18 – Implementing SportTracker in the Python SDK
Task 19 – Implementing RPCParDo in the Python SDK
Task 20 – Implementing SportTrackerMotivation in the Python SDK
Using the DataFrame API
Interactive programming using InteractiveRunner
Introducing and using cross-language pipelines
Summary
9
Section 3 Apache Beam: Advanced Concepts
Section 3 Apache Beam: Advanced Concepts
10
Chapter 7: Extending Apache Beam's I/O Connectors
chevron up
Chapter 7: Extending Apache Beam's I/O Connectors
Technical requirements
Defining splittable DoFn as a unification for bounded and unbounded sources
Task 21 – Implementing our own splittable DoFn – a streaming file source
Task 22 – A non-I/O application of splittable DoFn – PiSampler
The legacy Source API and the Read transform
Writing a custom data sink
Summary
11
Chapter 8: Understanding How Runners Execute Pipelines
Chapter 8: Understanding How Runners Execute Pipelines
Describing the anatomy of an Apache Beam runner
Explaining the differences between classic and portable runners
Understanding how a runner handles state
Exploring the Apache Beam capability matrix
Understanding windowing semantics in depth
Debugging pipelines and using Apache Beam metrics for observability
Summary
Why subscribe?
12
Other Books You May Enjoy
Other Books You May Enjoy
Packt is searching for authors like you
Share Your Thoughts

Defining splittable DoFn as a unification for bounded and unbounded sources

Beam offers a wide variety of source and sink transforms. We will not walk through them in this book because their details can be easily found online. In this book, we have used the KafkaIO transform heavily – other source and sink transforms are used analogously but specifically on the target storage system.

The question that arises is this: what should we do when there are either specific requirements for the way the data is read (or stored) or when we need to connect to a data source that Beam does not have a connector for? Let's first see how to implement a custom source.

A fundamental requirement for any source is that it has the ability to split itself. We need to split a bounded source in order to be able to parallelize its processing and we need to split an unbounded source to get a persistent moment in time we can return to in case of a failure. Such a moment in time is typically...

End of Section 3
Next Section
bookmark search playlist download
font-size

Change the font size

margin-width

Change margin width

day-mode

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Delete Bookmark

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete