A project to make Kafka easier to install on Kubernetes systems has been adopted by the Cloud Native Computing Foundation Technical Oversight Committee (TOC) as an incubating project.
Strimzi logo
Strimzi provides a set of operators to install and run the Kafka event processing platform on Kubernetes clusters, thus accruing all the benefits of K8s’ strengths in dynamic deployment.
“With Strimzi being part of the CNCF Incubator, I’m really optimistic that we can broaden the user base, the contributor base, and also the maintainers as a result of the progress we’re announcing today,” said Tom Bentley, founder and maintainer of the Strimzi project, in a statement.
“Running an Apache Kafka cluster on Kubernetes is not easy but since its inception, Strimzi has been acting as a game changer in this area,” further elaborated Paolo Patierno, founder and maintainer of the Strimzi project, in a statement. “The opportunity to deal with the day-2 operations burden by leveraging an operator-based approach, together with the declarative nature of Kubernetes by using custom resources, has been very well received.”
How to Run Kafka on Kubernetes
Originally developed at LinkedIn, Apache Kafka is rapidly becoming the de facto platform for horizontally scalable event-driven systems, often running real-time data operations.
In theory, it would make sense to run Kafka on Kubernetes, but this has proved to be very complicated in practice thus far.
“Deploying Kafka workloads on Kubernetes is not that simple, in general, because you will need to configure and run the Kafka brokers. And when you have to change anything on the brokers, like some configuration parameters, you have to start rolling all the brokers one by one,” Patierno said, in an interview with TNS. He also noted the software that manages metadata — originally Zookeeper though Kafka is migrating to the nimbler KRaft — must be managed as well.
Strimzi is not alone in supporting Kafka for Kubernetes. Confluent has a commercial plug-in for running Kafka on Kubernetes.
And from Cisco‘s incubation lab, there is KOperator, which bills itself as “Yet another Apache Kafka operator for Kubernetes.” This one also sidesteps stateful sets, by the use of Pods, ConfigMaps, and PersistentVolumeClaims.
Strimzi takes a different approach, Patierno explained, in that it allows you to define your cluster in terms of a Custom Resource Definition (CRD) that “extends the Kubernetes API in order to provide you a native way to interact with Kafka,” Patierno said. The CRD describes the configuration, the storage, number of brokers needed and other aspects. The CRD can then deploy the cluster needed. Change a parameter and the CRD will do an update.
Where Did Strimzi Come from?
Red Hat first released Strimzi in 2017, for the company’s OpenShift Kubernetes distribution, as the open source counterpart of AMQ Streams. The Operator pattern, created by CoreOS, provided the framework. CNCF took it on as a sandbox project in 2019.
Strimzi is composed of three operators:
-
The Cluster Operator deploys the Apache Kafka cluster by starting the brokers with the desired configuration, among other duties.
-
The Topic Operator creates, updates and deletes topics, on behalf of users, by using a KafkaTopic custom resource.
-
The User Operator provides cluster users to define permissions through ACLs and KafkaUser custom resources.
The Strimzi package also includes components for OAuth 2.0 protocol, and an HTTP-based endpoint to interact and configure a Kafka cluster.
Who Uses Strimzi
Today, the project now has more than 1,600 contributors from more than 180 organizations. It is being used in at least 15 (publicly disclosed) production environments, such as Axual, Decathlon, and SBB, among others.
Atruvia, the IT services arm of the German Cooperative Financial Group, has been using Strimzi for four years, to run Kafka on behalf of 150 teams, with a support team of only two and a half members. The cloud native design fits well with the company’s architecture, and Strimzi allows the team to stay current with the latest Kafka releases, according to a statement from the firm.
Sports gear provider Decathlon uses Strimzi to improve the company’s delivery times. “It meets our need to connect a large number of data systems with a high level of scalability, security and reliability,” said a spokesperson in a statement.
Colt McNealy, who is the leader of the technical staff of Kafka-based microservices provider LittleHorse has testified that Strimzi helped him get his start with Kafka when he knew little about managing a cluster.
“When running Kafka in production, dealing with cluster balancing, scaling, and monitoring is a daunting task, but the native Cruise Control integration with the KafkaRebalance CRD and the provided example metrics configurations (Grafana + Prometheus) made it possible for me as a Kafka novice to get something production-ready,” McNealy said, in a statement.
Cloud services provider Portworx has commented how much work has been done in Striumzi. “The project spent a considerable amount of time struggling to move beyond its beta stage, so many developers are still wary of using it in production,” wrote Portworx Technical Marketing Manager Ryan Wallner, in a blog post that has praised the progress the project has made since those early days.
Thus far, Strimzi has accumulated over 4,200 Stars on GitHub and has been integrated with other CNCF projects including Prometheus, OpenTelemetry, Helm, and others.
The project has its first virtual user conference coming up, StrimziCon, on May 22.
The post CNCF-Approved Strimzi Eases Apache Kafka onto K8s Clusters appeared first on The New Stack.
With a set of operators, Strimzi extends the Kubernetes API in order to provide a native way to interact with Kafka.