Kafka : Zookeeper, broker and consoles access for Producer and Consumer (lab)

kafka

Posted By: admin July 13, 2018

In this lab, we will work on setting up a Kafka environment, where a Producer and Consumer can communicate.

Prerequisite:

  • 2 computers/vm
  • Minimal installtion of CentOS 7 with updated kernel
  • SElinux and firewalld should be permanently disabled
  • Install “java-1.8.0-openjdk”
  • Install “wget”

Note : Please use ssh terminals as we will need to open multiple terminals.

In this lab we will be using “kafka_2.11-1.1.0“.

Create a folder “/Download”
[root@kafka1 /]# mkdir /Download

Go to the folder “/Download”
[root@kafka1 /]# cd /Download

Download kafka from “http://kafka.apache.org/”
[root@kafka1 Download]# wget http://www-eu.apache.org/dist/kafka/1.1.0/kafka_2.11-1.1.0.tgz

Extract the contents of the file.
[root@kafka1 Download]# tar -xvf kafka_2.11-1.1.0.tgz

Go to the subfolder “kafka_2.11-1.1.0” created now.
[root@kafka1 Download]# cd kafka_2.11-1.1.0

As explained in my previous tutorial, which can be accessed by clicking here, we can have multiple brokers working together as a cluster. The way these brokers identifies the brokers in the same cluster is via an application called zookeeper.

Now let us work on starting the zookeeper first. Ensure the terminal remains opened, and left alone till the end of the lab.
[root@kafka1 kafka_2.11-1.1.0]# bin/zookeeper-server-start.sh config/zookeeper.properties

If the zookeeper starts without any error, the next step is to start the broker node in the host. In case you would like to run multiple broker nodes in the same host, please read the other tutorial by clicking here. Open another ssh terminal window, and run the following command:
[root@kafka1 kafka_2.11-1.1.0]# bin/kafka-server-start.sh config/server.properties

As we know the Producer and the Customer in a communication should be using the same Topic. Now let us create a new Topic named “MyFirstTopic” that will be used in our sample communication.
[root@kafka1 kafka_2.11-1.1.0]# bin/kafka-console-consumer.sh –bootstrap-server localhost:9092 –topic MyFirstTopic

Let us check the status of the Topic now.
[root@kafka1 kafka_2.11-1.1.0]# bin/kafka-topics.sh –zookeeper localhost:2181 –describe –topic MyFirstTopic

Now the broker is ready and listening. Both Producer and Consumer can connect to the agreed Topic (queue) in the server. In this example we will be using the console applications to input from Producer and to see if the Consumer can receive it. In a real life scenario, the Producer, Broker and Consumer will be in three separate hosts. However, in our lab we will run all the three in the same node. As the broker is already running, let us start the console application for Producer.
[root@kafka1 kafka_2.11-1.1.0]# bin/kafka-console-producer.sh –broker-list localhost:9092 –topic MyFirstTopic

Now let us start the console application for Consumer in a new ssh terminal window:
[root@kafka1 kafka_2.11-1.1.0]# bin/kafka-console-consumer.sh –bootstrap-server localhost:9092 –topic MyFirstTopic

If everything went smooth without any error messages, go to the Producer’s console, and type “This is a test”. Can we see the same message appearing in the Consumer’s console !!!