Build Kafka Cluster with Kerberos

Reading time ~4 minutes

This is a for-beginner tutorial for those who already understand how Kafka works and the basic functionality of Kerberos.


  • Setup a simple pipeline for stream processing within the VMs.
  • Integrate Kafka with Kerberos’ SASL authentication.
  • Console pipeline to check if our setup works.

The versions of components in this post will be:

  • Ubuntu: 16.04.1
  • Kafka: with scala 2.11
  • Kerberos: 5

Prepare the Servers

We are going to have 3 Ubuntu servers in the VirtualBox:

server-kerberos A server for Kerberos. server-kafka A server for both Zookeeper and Kafka Broker. server-kafka-client A server for Kafka clients(producer and consumer).

You can also make server-kafka-client into two different servers, but the setups for these two servers will be exactly the same. To make things simple and clean, this post will only use one server to host both producer and consumer.

For the same reason, server-kafka can also be splited into two servers.

Installation of Kerberos and Kafka

Use VirtualBox to create all these 3 servers with Ubuntu. Go to your VirtualBox manager, and make sure all your boxes’ network adapters are set to NAT.

For server-kerberos:

To install kerberos, enter:

apt-get install krb5-admin-server krb5-kdc

During the installation, you will be asked for several settings. Enter the settings like below:

Default Kerberos version 5 realm? [VISUALSKYRIM]

Kerberos servers for your realm? []

Administrative server for your realm? []

For server-kafka, server-kafka-client:

Install krb5-user for SASL authentication:

sudo apt-get install krb5-user

During this installation, you will be asked the same questions. Just answer them with the same answer:

Default Kerberos version 5 realm? [VISUALSKYRIM]

Kerberos servers for your realm? []

Administrative server for your realm? []

These question will generate a Kerberos config file under you /etc/krb5.config.

Install kafka:

tar -xzf kafka_2.11-
cd kafka_2.11-

Later in this post, you will need to transfer authentication files(keytabs) between servers. For that purpose, this post will use scp, and openssh-server will be installed.

If you are going to use other methods to transfer files from server-kerberos, feel free to skip this installation.

apt-get install openssh-server

Setting up servers

Before starting to set up servers, we need to change our VMs’ network adapters to Host-Only and reboot to get an individual IP address for each VM.

After that, go to each server to get their IP address by


Then input those IP address and the hostnames into /etc/hosts. Something like this:

Then make sure all these 3 servers can ping each other by using the hostname.

After that, we start to set up our servers one by one:

Kerberos Server

Create the new realm.

sudo krb5_newrealm

This might stuck at the place where command-line prompts Loading random data.

If this happens, run the following code first: cat /dev/sda > /dev/urandom

Then edit /etc/krb5.conf.

The updated content should be like:

    default_realm = VISUALSKYRIM


        kdc =
        admin_server =


[domain_realm] = VISUALSKYRIM

Next, add principals for each of your roles:

- zookeeper
- kafka
- kafka-client


$ sudo kadmin.local

> addprinc zookeeper
> ktadd -k /tmp/zookeeper.keytab zookeeper
> addprinc kafka
> ktadd -k /tmp/kafka.keytab kafka
> addprinc kafka-client
> ktadd -k /tmp/kafka-client.keytab kafka-client

Move /tmp/zookeeper.keytab and /tmp/kafka.keytab to your server-kafka, and move /tmp/kafka-client.keytab to your server-kafka-client.

Kafka Server

Just like real world, every individual(program) in the distributed system must tell the authority(Kerberos) two things to identify itself:

The first thing is the accepted way for this role to be identified. It’s like in America, people usually use drive license, and in China people use ID card, while in Japan, people use so called MyNumber card.

The second thing is the file or document that identifies you according to your accepted identify method.

The file to identify the role in our SASL context, is the keytab file we generated via the kadmin just now.

Suggest you put zookeeper.keytab and kafka.keytab under /etc/kafka/ of you server-kafka.

We need a way to tell our program where to find this file and how to hand it over to the authority(Kerberos). And that will be the JAAS file.

We create the JAAS files for Zookeeper and Kafka and put it to /etc/kafka/zookeeper_jaas.conf and /etc/kafka/kafka_jaas.conf.

Server { required debug=true
KafkaServer { required debug=true

// For Zookeeper Client
Client { required debug=true

To specify the locations of these JAAS file, we need to put the locations into JVM options like:


To make this post easy and simple, I choose to modify the the bin/, bin/ and bin/ to insert those JVM options into the launch command.

To enable SASL authentication in Zookeeper and Kafka broker, simply uncomment and edit the config files config/ and config/

For config/


For config/


Then start the Zookeeper and Kafka by:

$ bin/ config/
$ bin/ config/

Kafka Client Server

The setting up for you server-kafka-client is quite similar to what you’ve just done for server-kafka.

For JAAS file, because we are going to use the same principal and keytab for both producer and consumer in this case, we only need to create one single JAAS file /etc/kafka/kafka_client_jaas.conf:

KafkaClient { required debug=true

We also need to put JVM options to the bin/

Give it a try

Now we can check if our setup actually works.

First we start a console-producer:

bin/ --broker-list --topic test \
--producer-property security.protocol=SASL_PLAINTEXT \
--producer-property sasl.mechanism=GSSAPI \

And start a console-comsumer:

bin/ --bootstrap-server --topic test \
--consumer-property security.protocol=SASL_PLAINTEXT \
--consumer-property sasl.mechanism=GSSAPI \

Then input some message into the console-producer to see if the same message prompted in console-consumer after a few seconds.