Let us have a quick introduction to Apache Kafka in this article. Kafka is a powerful real-time data exchange system that is used in many large companies : An event streaming platform used to collect, process, store, and integrate data at scale. It has numerous use cases including distributed streaming, stream processing, data integration, and pub/sub messaging.

GET KAFKA

First, we need to install Kafka on our computer. Kafka is an open-source software that we can download from the official website www.kafka.apache.org. There you will find the latest version for download.

START THE KAFKA ENVIRONMENT

After we have installed Kafka, we can start the environment. For this, we open a terminal window and execute the following commands:

bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties

These commands start the Zookeeper service and the Kafka broker service, which are required for the operation of Kafka.

CREATE A TOPIC

we create a topic to store our events. A topic is like a database table where we can store data. We can create a new topic with the following command:

bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic my-topic

This command creates a new topic called “my-topic” that we can use to store our events.

All of Kafka’s command line tools have additional options: run the kafka-topics.sh command without any arguments to display usage information. For example, it can also show you details such as the partition count of the new topic:

$ bin/kafka-topics.sh --describe --topic my-topic --bootstrap-server localhost:9092

Topic: quickstart-events        TopicId: NPmZHyhbR9y00wMglMH2sg PartitionCount: 1       ReplicationFactor: 1	Configs:
    Topic: quickstart-events Partition: 0    Leader: 0   Replicas: 0 Isr: 0

WRITE EVENTS

Now we can write events to our new topic. For this, we open another terminal command and execute the following command:

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic

//each line we enter will result in a separate event being written to the topic.
This is my first event
This is my second event

This opens an input window where we can enter events. Each line we enter will be written as an (separate) event into the topic.

PS: A Kafka client communicates with the Kafka brokers via the network for writing (or reading) events. Once received, the brokers will store the events in a durable and fault-tolerant manner for as long as you need—even forever.

We can stop the producer client with Ctrl-C at any time.

READ EVENTS

To read the events from the topic, we open another terminal command and execute the following command:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --from-beginning

This command starts a Kafka consumer that reads all events from the “my-topic” topic and outputs them in our terminal window.

IMPORT/EXPORT DATA WITH KAFKA CONNECT

Kafka Connect is a tool that allows us to import data from other systems into Kafka or export data from Kafka to other systems. This allows us to easily and efficiently integrate large amounts of data into Kafka.

PROCESS EVENTS WITH KAFKA STREAMS

Kafka Streams is a library that allows us to process and transform events in real-time. This allows us to perform complex data analyses and transformations directly in Kafka.

TERMINATE THE KAFKA ENVIRONMENT

When we are done using Kafka, we can shut down the environment again. For this, we execute the following commands:

bin/kafka-server-stop.sh
bin/zookeeper-server-stop.sh

These commands properly stop the Kafka broker and Zookeeper services.