In this chapter, we developed Apache Kafka producer and consumer applications and utilized Spark's Structured Streaming engine to process streaming data consumed from a Kafka topic. In our real-world case study, we designed, developed, and deployed an end-to-end stream processing pipeline that was capable of consuming real tweets being authored across the world and then classified their underlying sentiment using machine learning, all of which was done in real time.

In this book, we went on both a theoretical and a hands-on journey through some of the most important and exciting technologies and frameworks that underpin the data-intelligence-driven revolution being seen across industry today. We started out by describing a new breed of distributed and scalable technologies that allow us to store, process, and analyze huge volumes of structured, semi-structured, and...