Running Kafka on Kubernetes with Strimzi

Kafka powers real-time data pipelines for tech giants, processing millions of events per second with ease. As a leading distributed event streaming platform, Kafka delivers unmatched high-throughput messaging capabilities.

However, as Uncle Ben wisely noted:
"With great power comes great responsibility".

That’s where Strimzi comes in. This open-source project transforms Kafka deployment and management on Kubernetes from a complex endeavor into a streamlined process. In this post, we’ll explore how Strimzi simplifies Kafka operations and why it might be the perfect addition to your event-driven architecture.

Why Strimzi Matters in Production

While this playground uses ephemeral storage for simplicity, production deployments require careful planning:

Persistent Storage: Use fast, local SSDs for brokers to ensure low latency and high throughput (60k IOPS+ the faster, the better).
Network Configuration: Each broker needs direct addressing via headless services. Network throughput is critical.
Resource Isolation: Dedicate nodes to Kafka brokers using taints/tolerations and node selectors. Production brokers often run on nodes with 64GB+ RAM and 16+ CPU cores.
KRaft Mode: This demo uses KRaft mode (no Zookeeper), which simplifies operations and is the future of Kafka.
Rack Awareness: For high availability, spread brokers across availability zones/racks.
Monitoring & Security: Integrate with Prometheus/Grafana and enforce TLS/OAuth2 for secure, observable clusters.

Strimzi makes it easy to manage these configurations through custom resources, allowing you to focus on building your applications rather than managing infrastructure. It uses the Kubernetes Operator pattern to watch for Kafka-related custom resources and manage the underlying StatefulSets, Services, and secrets for you. This means you can declaratively manage your Kafka infrastructure just like any other Kubernetes resource.

Prerequisites

Any Kubernetes cluster (e.g., Kind, Minikube, AKS, EKS, GKE) will work, but this guide uses Kind.
- Docker installed and running (for Kind)
- Kind installed (Kind installation)
kubectl installed (kubectl installation)
Helm installed (Helm installation)
OpenSSL and Keytool (for certificate generation)

(Optional: Install k9s for a better Kubernetes CLI experience.)

Quick Start

In the following steps, we will setup a local Kafka cluster using Strimzi on Kind. The authentication will be done using mTLS, and we will deploy Kafdrop (a web UI for Kafka) to visualize the cluster.

Clone this repository

git clone https://github.com/gjorgji-ts/kafka-playground.git
cd kafka-playground

Create the Kind cluster and deploy Strimzi operator
```
chmod +x setup-cluster.sh
./setup-cluster.sh
```

Deploy the Kafka cluster

kubectl apply -f kafka/kafka-kraft.yaml
kubectl wait kafka/demo --for=condition=Ready --timeout=300s -n kafka

Create Kafdrop user and certificates

Create a user for Kafdrop with mTLS authentication:

kubectl apply -f kafka/kafdrop-mtls-user.yaml

Validate that the secret was created (if not, give it a few seconds):

kubectl get secret kafdrop-mtls-user -n kafka

Extract the certificates:

mkdir -p certs
kubectl get secret kafdrop-mtls-user -n kafka -o jsonpath='{.data.user\.crt}' | base64 -d > certs/kafdrop-mtls-user.crt
kubectl get secret kafdrop-mtls-user -n kafka -o jsonpath='{.data.user\.key}' | base64 -d > certs/kafdrop-mtls-user.key
kubectl get secret demo-cluster-ca-cert -n kafka -o jsonpath='{.data.ca\.crt}' | base64 -d > certs/ca.crt

Create PKCS12 keystore for Kafdrop

openssl pkcs12 -export \
  -in certs/kafdrop-mtls-user.crt \
  -inkey certs/kafdrop-mtls-user.key \
  -out certs/kafdrop-mtls-user.p12 \
  -name kafdrop-mtls-user \
  -CAfile certs/ca.crt \
  -caname root \
  -passout pass:< A strong password >

Create a secret for Kafdrop with the PKCS12 keystore and CA certificate

kubectl create secret generic kafdrop-mtls-cert-secret \
  --from-file=kafdrop-mtls-user.p12=certs/kafdrop-mtls-user.p12 \
  --from-file=ca.crt=certs/ca.crt \
  -n kafka

Configure Kafdrop deployment

Create the base64 encoded value for kafka.properties:

echo -n "
security.protocol=SSL
ssl.keystore.type=PKCS12
ssl.truststore.type=PEM
ssl.endpoint.identification.algorithm=""
ssl.keystore.location=/certs/kafdrop-mtls-user.p12
ssl.keystore.password=< The same password used above >
ssl.truststore.location=/certs/ca.crt
" | base64

Replace the base64 value in kafka/kafdrop-mtls.yaml under the KAFKA_PROPERTIES environment variable.

Deploy Kafdrop

kubectl apply -f kafka/kafdrop-mtls.yaml

Get the node IP address
```
kubectl get nodes -o wide
```
Access Kafdrop at http://< Any Node IP >:30000 in your browser.

Creating a Kafka Topic

Once your Kafka cluster is up and running, you can start creating topics. Topics are the fundamental unit of organization in Kafka, allowing you to categorize and manage your data streams effectively. Some key points about Kafka topics:

Topics are multi-subscriber, meaning multiple consumers can read from the same topic simultaneously.
Topics are partitioned, allowing for parallel processing and scalability.
Topics are replicated across multiple brokers for fault tolerance and high availability.

To create a topic in your Kafka cluster, you can use the KafkaTopic custom resource. This example creates a topic with 2 partitions and 3 replicas, ensuring high availability.

We have a KafkaTopic manifest in kafka/kafka-topic.yaml:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: demo-topic
  labels:
    strimzi.io/cluster: demo
spec:
  partitions: 2
  replicas: 3
  config:
    retention.ms: 7200000 # How long to keep messages (e.g., 2 hours)
    segment.bytes: 1073741824 # How large each segment file can be (e.g., 1GB)

Apply the topic manifest

kubectl apply -f kafka/kafka-topic.yaml

Verify the topic creation
```
kubectl get kafkatopic demo-topic -n kafka
```
You should see the topic created with the specified partitions and replicas.
Check the topic in Kafdrop
- Open Kafdrop in your browser at http://<any-node-ip>:30000
- Under the “Topics” section, you should see demo-topic listed with its partitions and replicas.

Best Practices:

Maintain at least 3 replicas for high availability.
Set min.insync.replicas to 2 for safety (under the config section), ensuring that at least 2 replicas are in sync before acknowledging a write.
Use more partitions for higher parallelism.
Monitor partition distribution with the Strimzi KafkaRebalance CRD.

Scaling the Kafka Brokers

You can easily scale the number of Kafka broker pods in your local Strimzi playground by editing the replicas field in your Kafka cluster manifest.

Edit the broker count

Open kafka/kafka-kraft.yaml and find the replicas field in the KafkaNodePool section for the broker role:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: broker
  labels:
    strimzi.io/cluster: demo
spec:
  replicas: 3 # Change this to your desired number of replicas
  roles:
    - broker
  storage:
    type: ephemeral

For example, to scale to 5 brokers, set replicas: 5.

Apply the updated manifest
```
kubectl apply -f kafka/kafka-kraft.yaml
```
Strimzi will automatically add or remove broker pods to match the new replica count.
Verify the broker pods
```
kubectl get pods -n kafka -l strimzi.io/cluster=demo
```
You should see the number of demo-kafka- pods match your desired replica count.

Production Scaling Tips:

Always scale in multiples of your replication factor for optimal partition placement.
Use the KafkaRebalance CRD after scaling to optimize partition distribution.
Monitor disk I/O during partition rebalancing.
Consider vertical scaling (CPU/RAM) before adding more brokers for small clusters.

Note: For advanced scenarios like Kafka Connect, MirrorMaker, refer to the Strimzi documentation.

Cleanup

To delete the Kind cluster and all resources:

kind delete clusters kafka-playground

With this playground, you can experiment confidently with Kafka on Kubernetes right from your local machine. Adjust the manifests, test different security configurations, and observe how the Strimzi Operators manage cluster operations seamlessly. Enjoy your Kafka journey! 🚀