In this chapter, we discussed the concepts of the stream processing systems, Spark streaming, DStreams of Apache Spark, what DStreams are, DAGs and lineages of DStreams, Transformations, and Actions. We also looked at window concept of stream processing. We also looked at a practical examples of consuming tweets from Twitter using Spark Streaming.

In addition, we looked at receiver-based and direct stream approaches of consuming data from Kafka. In the end, we also looked at the new structured streaming, which promises to solve many of the challenges such as fault tolerance and exactly once semantics on the stream. We also discussed how structured streaming also simplifies the integration with messaging systems such as Kafka or other messaging systems.

In the next chapter, we will look at graph processing and how it all works.