IBM Support

Log Analysis - Basic Maintenance Tasks of a Kafka Cluster

Technical Blog Post


Abstract

Log Analysis - Basic Maintenance Tasks of a Kafka Cluster

Body

If you have plans to expand the Kafka cluster of your Log Analysis setup or had already done so, it is always good to know how to manage your Kafka cluster properly.

 

There are certainly a few important tasks when it comes to managing a Kafka cluster:

(1) How many brokers are connected?

(2) What are the topics available?

(3) How do you create new topic?

(4) How do you check the content of an existing topic?

 

While management tasks are not limited to those listed above, but we will focus on those as a starter.

 

(1) How many brokers are connected?

To know how many Kafka brokers are currently connected, it is quite simple.

 

[danielyeap@kafka1 bin]$ ./zookeeper-server-start.sh -daemon ../config/zookeeper.properties 
[danielyeap@kafka1 bin]$ ./zookeeper-shell.sh kafka1:2181

...

ls /
[cluster, controller_epoch, brokers, zookeeper, admin, isr_change_notification, consumers, log_dir_event_notification, latest_producer_id_block, config]
ls /brokers
[ids, topics, seqid]
ls /brokers/ids
[]             ** A blank array means NO Kafka brokers is connected

ls /brokers/ids
[0, 1]       ** This means 2 Kafka brokers are connected with ID 0 and 1 respectively

 

 

(2) What are the topics available?

To examine what are the topics available within the Kafka cluster, you can perform the following.

 

[danielyeap@kafka1 bin]$ ./zookeeper-shell.sh kafka1:2181
...

ls /brokers/topics
[scala_access_log, aixerr, iis, oracle, test, oracle2, web_access_log, winevt, __consumer_offsets]

 

 

(3) How do you create new topic?

To create a new topic, you can use the "kafka-topics.sh" tool provided with Kafka.

 

[danielyeap@kafka1 bin]$ pwd
/opt/kafka_2.11-0.10.0.0/bin
[danielyeap@kafka1 bin]$ ls -l *topic*
-rwxr-xr-x 1 danielyeap danielyeap 863 May 18  2016 kafka-topics.sh
[danielyeap@kafka1 bin]$ ./kafka-topics.sh --create --topic blogtest --partitions 2 --zookeeper kafka1:2181 --replication-factor 1
...
Created topic "blogtest".
[danielyeap@kafka1 bin]$ ./kafka-topics.sh --list --zookeeper kafka1:2181
...
__consumer_offsets
aixerr
blogtest
iis
oracle
oracle2
scala_access_log
test
web_access_log
winevt
[danielyeap@kafka1 bin]$ 

 

(4) How do you check the content of an existing topic?

To view the content of a particular topic, you can utilize the "kafka-console-consumer.sh" tool provided.

 

[danielyeap@kafka1 bin]$ ./kafka-console-consumer.sh --topic test --from-beginning --zookeeper kafka1:2181
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Hello World!
I am fine!
How are you?
Bye!

 

Have fun Kafka-ing!

 

 

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"","label":""},"Component":"","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"","label":""}}]

UID

ibm11081665

Page Feedback