-
Book Overview & Buying
-
Table Of Contents
-
Feedback & Rating

Building Big Data Pipelines with Apache Beam
By :

This task will be a reimplementation of Task 8 from Chapter 3, Implementing Pipelines using Stateful Processing. We will use a stateful DoFn
to batch the elements for a defined maximal amount of time. As always, we will restate the problem again for clarity.
Use a given RPC service to augment data in the input stream using batched RPCs with batches whose size are about K elements. Also, resolve the batch after, at most, time T to avoid (possibly) an infinitely long waiting time for elements in small batches.
As in the previous task, we will skip the discussion of the problem's decomposition as we discussed that when we implemented Task 8. Instead, we will jump directly into its implementation using the Python SDK.
The implementation can be found in chapter6/src/main/python/rpc_par_do.py
. It can be broken down into the RPCParDoStateful
transform, which is declared...