Zookeeper & Kafka Install : Single node and multiple brokers

bogotobogo.com site search:

Note

In the previous chapter (Zookeeper & Kafka Install : Single node and single broker), we run Kafka and Zookeeper with single broker. Now we want to setup a Kafka cluster with multiple brokers as shown in the picture below:

Picture source: Learning Apache Kafka 2nd ed.

The more brokers we add, more data we can store in Kafka. In terms of resources, Kafka is typically IO bound. Performance will be limited by disk speed and file system cache - good SSD drives and file system cache can easily allow millions of messages/sec to be supported per second.

Starting Zookeeper

To launch Zookeeper, we'll use the default configuration that Kafka provides:

~/kafka/config$ ls
connect-console-sink.properties    connect-file-source.properties  log4j.properties        zookeeper.properties
connect-console-source.properties  connect-log4j.properties        producer.properties
connect-distributed.properties     connect-standalone.properties   server.properties
connect-file-sink.properties       consumer.properties             tools-log4j.properties

Let's start the local Zookeeper instance:

$ bin/zookeeper-server-start.sh config/zookeeper.properties

zookeeper.properties

This remains the setup as in the previous chapter (Zookeeper & Kafka Install : A single node and a single broker cluster).

Kafka provides us with the required property files which defining minimal properties required for a single broker-single node cluster:

# the directory where the snapshot is stored.
dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0

The Zookeeper, by default, will listen on *:2181/tcp.

Starting Kafka broker

To setup multiple brokers on a single node, different server property files are required for each broker. Each property file defines different values for the following properties:

broker.id
port
log.dir

So, for broker_1 will use server_1.properties and broker_2 will use server_2.properties ass shown below.

For broker_1:

broker.id=1
port=9093
log.dir=/tmp/kafka-logs-1

and for broker_2:

broker.id=2
port=9094
log.dir=/tmp/kafka-logs-2

Now, we want to start each new broker in a separate console window:

$ bin/kafka-server-start.sh config/server_1.properties
$ bin/kafka-server-start.sh config/server_2.properties

Note that we already have one broker that's running (broker.id=0, port=9092, log.dir=/tmp/kafka-logs).

Creating a Kafka topic

We can create topics on the Kafka server. Here, we'll create a topic named "replica-kafkatopic" with with a replication factor of three. The following kafka-topics.sh will create a topic:

$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partition 1 --topic replica-kafkatopic
Created topic "replica-kafkatopic".

To get a list of topics, we can use "--list -- ..." command:

$ bin/kafka-topics.sh --list --zookeeper localhost:2181
kafkatopic
replica-kafkatopic

Starting a producer - sending messages

If we use a single producer to get connected to all the brokers, we need to pass the initial list of brokers. The information of the remaining brokers is identified by querying the broker passed within broker-list:

The producer client can accept inputs from the command line and publishes them as a message to the Kafka cluster. Each new line entered, by default, is a new message as shown below:

$ bin/kafka-console-producer.sh --broker-list localhost:9092 localhost:9093 localhost:9094 --topic replica-kafkatopic
Welcom to Kafka, again.
This is our first multiple broker cluster.

Starting a consumer - consuming messages

Consumer client consumes messages, and we'll use the same consumer client:

$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic replica-kafkatopic
Welcome to Kafka, again.
This is our first single broker cluster.