Streaming data into Kafka is fairly easy. There are producers that send device-to-cloud messages and consumers that receive cloud-to-device messages. In the following example, we are going to implement a producer:
- Download an example project:
git clone https://github.com/Microshak/KafkaWeatherStreamer.git
cd KafkaWeatherStreamer
- Install the requirements:
pip install -r requirements.txt
- Run the weather.py file:
python3 weather.py
You should now be able to look at your Kafka Control Center and see data flowing in. The Kafka Streams API is a real-time platform that can perform ML computations with millisecond latency. The Streams API has the concepts of KTables and KStreams. KStreams are data streaming into Kafka on various topics. KTables are streams turned into tables where the data is updated every time there is a new record associated with its primary key. This allows multiple streams to be joined together similarly to how tables in a database are joined together...