{"id":3257,"date":"2024-05-15T16:47:54","date_gmt":"2024-05-15T14:47:54","guid":{"rendered":"https:\/\/nguenkam.com\/blog\/?p=3257"},"modified":"2024-05-16T09:59:27","modified_gmt":"2024-05-16T07:59:27","slug":"apache-kafka-quickstart","status":"publish","type":"post","link":"https:\/\/nguenkam.com\/blog\/index.php\/2024\/05\/15\/apache-kafka-quickstart\/","title":{"rendered":"Apache KAFKA – Quickstart"},"content":{"rendered":"\n

Let us have a quick introduction to Apache Kafka in this article. Kafka is a powerful real-time data exchange system that is used in many large companies : An event streaming platform used to collect, process, store, and integrate data at scale. It has numerous use cases including distributed streaming, stream processing, data integration, and pub\/sub messaging.<\/em><\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

GET KAFKA<\/h4>\n\n\n\n

First, we need to install Kafka on our computer. Kafka is an open-source software that we can download from the official website www.kafka.apache.org. There you will find the latest version for download.<\/p>\n\n\n\n

START THE KAFKA ENVIRONMENT<\/h4>\n\n\n\n

After we have installed Kafka, we can start the environment. For this, we open a terminal window and execute the following commands:<\/p>\n\n\n\n

bin\/zookeeper-server-start.sh config\/zookeeper.properties\nbin\/kafka-server-start.sh config\/server.properties\n<\/code><\/pre>\n\n\n\n
These commands start the Zookeeper service and the Kafka broker service, which are required for the operation of Kafka.<\/p>\n\n\n\n
CREATE A TOPIC <\/h4>\n\n\n\n
we create a topic to store our events. A topic is like a database table where we can store data. We can create a new topic with the following command:<\/p>\n\n\n\n
bin\/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic my-topic\n<\/code><\/pre>\n\n\n\n
This command creates a new topic called “my-topic” that we can use to store our events.<\/p>\n\n\n\n
All of Kafka’s command line tools have additional options: run the kafka-topics.sh<\/code> <\/strong><\/em>command without any arguments to display usage information. For example, it can also show you details such as the partition count<\/a> of the new topic:<\/p>\n\n\n\n
$ bin\/kafka-topics.sh --describe --topic my-topic --bootstrap-server localhost:9092\n\nTopic: quickstart-events        TopicId: NPmZHyhbR9y00wMglMH2sg PartitionCount: 1       ReplicationFactor: 1\tConfigs:\n    Topic: quickstart-events Partition: 0    Leader: 0   Replicas: 0 Isr: 0<\/code><\/pre>\n\n\n\n
WRITE EVENTS<\/h4>\n\n\n\n
Now we can write events to our new topic. For this, we open another terminal command and execute the following command:<\/p>\n\n\n\n
bin\/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic\n\n\/\/each line we enter will result in a separate event being written to the topic.\nThis is my first event\nThis is my second event\n<\/code><\/pre>\n\n\n\n
This opens an input window where we can enter events. Each line we enter will be written as an (separate) event into the topic.<\/p>\n\n\n\n
PS<\/strong>: A Kafka client communicates with the Kafka brokers via the network for writing (or reading) events. Once received, the brokers will store the events in a durable and fault-tolerant manner for as long as you need\u2014even forever.<\/p>\n\n\n\n
We can stop the producer client with Ctrl-C<\/code> <\/em><\/strong>at any time.<\/p>\n\n\n\n
READ EVENTS<\/h4>\n\n\n\n
To read the events from the topic, we open another terminal command and execute the following command:<\/p>\n\n\n\n
bin\/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --from-beginning\n<\/code><\/pre>\n\n\n\n
This command starts a Kafka consumer that reads all events from the “my-topic” topic and outputs them in our terminal window.<\/p>\n\n\n\n
IMPORT\/EXPORT DATA WITH KAFKA CONNECT<\/h4>\n\n\n\n
Kafka Connect<\/strong> is a tool that allows us to import data from other systems into Kafka or export data from Kafka to other systems. This allows us to easily and efficiently integrate large amounts of data into Kafka.<\/p>\n\n\n\n
PROCESS EVENTS WITH KAFKA STREAMS<\/h4>\n\n\n\n
Kafka Streams<\/strong> is a library that allows us to process and transform events in real-time. This allows us to perform complex data analyses and transformations directly in Kafka.<\/p>\n\n\n\n
TERMINATE THE KAFKA ENVIRONMENT<\/h4>\n\n\n\n
When we are done using Kafka, we can shut down the environment again. For this, we execute the following commands:<\/p>\n\n\n\n
bin\/kafka-server-stop.sh\nbin\/zookeeper-server-stop.sh\n<\/code><\/pre>\n\n\n\n
These commands properly stop the Kafka broker and Zookeeper services.<\/p>\n","protected":false},"excerpt":{"rendered":"
Let us have a quick introduction to Apache Kafka in this article. Kafka is a powerful real-time data exchange system that is used in many large companies : An event streaming platform used to collect, process, store, and integrate data at scale. It has numerous use cases including distributed streaming, stream processing, data integration, and […]<\/p>\n","protected":false},"author":1,"featured_media":3258,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[847],"tags":[840,851,852,848,850,849],"_links":{"self":[{"href":"https:\/\/nguenkam.com\/blog\/index.php\/wp-json\/wp\/v2\/posts\/3257"}],"collection":[{"href":"https:\/\/nguenkam.com\/blog\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nguenkam.com\/blog\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nguenkam.com\/blog\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nguenkam.com\/blog\/index.php\/wp-json\/wp\/v2\/comments?post=3257"}],"version-history":[{"count":4,"href":"https:\/\/nguenkam.com\/blog\/index.php\/wp-json\/wp\/v2\/posts\/3257\/revisions"}],"predecessor-version":[{"id":3266,"href":"https:\/\/nguenkam.com\/blog\/index.php\/wp-json\/wp\/v2\/posts\/3257\/revisions\/3266"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/nguenkam.com\/blog\/index.php\/wp-json\/wp\/v2\/media\/3258"}],"wp:attachment":[{"href":"https:\/\/nguenkam.com\/blog\/index.php\/wp-json\/wp\/v2\/media?parent=3257"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nguenkam.com\/blog\/index.php\/wp-json\/wp\/v2\/categories?post=3257"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nguenkam.com\/blog\/index.php\/wp-json\/wp\/v2\/tags?post=3257"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}