Installation of Apache Kafka 3.9.1 on Azure VM (Ubuntu 24.04)

Overview

  • This guide explains how to install and configure Kafka 3.9.1 on an Azure Ubuntu VM in KRaft mode (no ZooKeeper required). Kafka 3.9.x supports both ZooKeeper and KRaft, but new deployments should use KRaft.

Download & extract Kafka

cd /tmp
wget https://downloads.apache.org/kafka/3.9.1/kafka_2.13-3.9.1.tgz
sudo mkdir -p /opt/kafka
sudo tar -xzf kafka_2.13-3.9.1.tgz -C /opt/kafka --strip-components=1
sudo useradd -m -s /bin/bash kafka
sudo chown -R kafka:kafka /opt/kafka

  • Explanation

    • wget → download Kafka binary

    • /opt/kafka → standard installation directory

    • useradd → creates dedicated kafka user (security)

    • chown → ensures Kafka user owns the files

Create Kafka data directory

sudo mkdir -p /opt/kafka/data
sudo chown -R kafka:kafka /opt/kafka/data

  • Kafka stores logs, topics, and messages here.

Configure Kafka (KRaft mode)

  • Update the server.properties file using the below commands.

sudo -u kafka nano /opt/kafka/config/kraft/server.properties

  • Go through the file and add or update the following configuration accordingly.

process.roles=broker,controller
node.id=1
controller.quorum.bootstrap.servers=localhost:9093

listeners=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
advertised.listeners=PLAINTEXT://<YOUR_PUBLIC_IP>:9092,CONTROLLER://<YOUR_PUBLIC_IP>:9093

inter.broker.listener.name=PLAINTEXT
log.dirs=/opt/kafka/data

👉 Replace <YOUR_PUBLIC_IP> with your VM’s public IP from the Azure portal.

  • Save and exit using (Ctrl+X → Press Y → Enter).

Format Kafka storage

  • Generate a cluster ID.

cd /opt/kafka
sudo -u kafka ./bin/kafka-storage.sh random-uuid

  • Copy the UUID and format the storage.

sudo -u kafka ./bin/kafka-storage.sh format -t <UUID> -c /opt/kafka/config/kraft/server.properties

👉 Replace <UUID> with your generated random UUID.

Foreground test

  • To start the kafka in the foreground, run the following commands.

sudo -u kafka /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/kraft/server.properties

  • Runs Kafka in the foreground (for testing).

  • Stop with Ctrl+C.

Run Kafka as a Service

  • Create a systemd service.

sudo tee /etc/systemd/system/kafka.service <<EOF
[Unit]
Description=Apache Kafka 3.9.1 (KRaft)
After=network.target

[Service]
Type=simple
User=kafka
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/kraft/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh /opt/kafka/config/kraft/server.properties
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

  • Enable and start by running the following commands.

sudo systemctl daemon-reload
sudo systemctl enable kafka
sudo systemctl start kafka
sudo systemctl status kafka

Open ports

  • In Azure Portal

    • Add inbound rules for

      • 9092 (Kafka broker)

      • 9093 (KRaft controller)

sudo ufw allow 9092/tcp
sudo ufw allow 9093/tcp

Test Kafka

  • Create a topic - test-topic

sudo -u kafka /opt/kafka/bin/kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

  • Produce a message - hello from kafka

echo "hello from kafka" | sudo -u kafka /opt/kafka/bin/kafka-console-producer.sh --topic test-topic --bootstrap-server localhost:9092

  • Consume messages

sudo -u kafka /opt/kafka/bin/kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server localhost:9092

Final notes

  • Kafka is now running in KRaft mode without ZooKeeper.

  • Your Spring Boot applications can connect with

  • For every service use this syntax for enabling kafka.

spring:
  kafka:
    bootstrap-servers: <YOUR_PUBLIC_IP>:9092