We are delighted to announce the new Strimzi 0.11.1 release with some new features!
JBOD storage support for Kafka brokers
Other than supporting the ephemeral and persistent storage, Strimzi now supports JBOD (Just a Bunch of Disks) storage for Kafka brokers (but not for Zookeeper nodes).
The JBOD data storage configuration provides a way for Kafka brokers to make use of multiple disks. JBOD is one approach to providing increased data storage for Kafka brokers. It can also improve performance when there is an independent I/O path for each disk. A JBOD configuration is described by one or more volumes, each of which can be either ephemeral or persistent.
To use JBOD with Strimzi, the storage type must be set to the new jbod value.
Then the volumes property allows you to describe the disks that make up your JBOD storage array or configuration.
Here’s an example snippet with a JBOD configuration.
apiVersion: kafka.strimzi.io/v1alpha1
kind: Kafka
metadata:
name: my-cluster
spec:
kafka:
...
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
size: 100Gi
deleteClaim: false
- id: 1
type: persistent-claim
size: 100Gi
deleteClaim: false
The Cluster Operator will configure the log.dirs Kafka parameter accordingly, in order to store messages logs on the different volumes.
Improved configuration for off-cluster access
This improvement was mostly driven and contributed by the community! A lot of developers asked about having more flexibility and extensibility on the listeners configuration for exposing the Kafka cluster outside of the Kubernetes/OpenShift cluster. Thanks to djotanov for opening a pull request about part of these new features.
When configuring the listeners.external section of the Kafka resource, it’s now possible to leverage a new overrides property for a more powerful configuration.
For example, by default, using a route listener, the route hostnames are automatically assigned by OpenShift.
Now, you can override the assigned route hostnamess, specifying the names you want to use.
Here’s an example snippet for overriding route hosts.
apiVersion: kafka.strimzi.io/v1alpha1
kind: Kafka
metadata:
name: my-cluster
spec:
kafka:
...
listeners:
external:
type: route
authentication:
type: tls
overrides:
bootstrap:
host: bootstrap.myrouter.com
brokers:
- broker: 0
host: broker-0.myrouter.com
- broker: 1
host: broker-1.myrouter.com
- broker: 2
host: broker-2.myrouter.com
The nodeport listener is now also more configurable.
By default, the port numbers used for the bootstrap and broker services are automatically assigned by OpenShift or Kubernetes.
This new Strimzi release lets you can override the assigned node ports by specifying the requested port numbers.
Here’s an example snippet for overriding node ports.
apiVersion: kafka.strimzi.io/v1alpha1
kind: Kafka
metadata:
name: my-cluster
spec:
kafka:
...
listeners:
external:
type: nodeport
tls: true
authentication:
type: tls
overrides:
bootstrap:
nodePort: 32100
brokers:
- broker: 0
nodePort: 32000
- broker: 1
nodePort: 32001
- broker: 2
nodePort: 32002
Finally, a really interesting new feature on listeners is customization of the Kafka brokers’ advertised addresses. By default, Strimzi tries to automatically determine the hostnames and ports that your Kafka cluster advertises to its clients. This is not sufficient in all situations, because the infrastructure on which Strimzi is running might not provide the right hostname or port through which Kafka can be accessed. Now, you can customize the advertised hostname and port. Strimzi will then automatically configure the advertised address in the Kafka brokers and add it to the broker certificates so it can be used for TLS hostname verification.
Here’s an example snippet for overriding advertised addresses.
apiVersion: kafka.strimzi.io/v1alpha1
kind: Kafka
metadata:
name: my-cluster
spec:
kafka:
...
listeners:
external:
type: route
authentication:
type: tls
overrides:
brokers:
- broker: 0
advertisedHost: example.hostname.0
advertisedPort: 12340
- broker: 1
advertisedHost: example.hostname.1
advertisedPort: 12341
- broker: 2
advertisedHost: example.hostname.2
advertisedPort: 12342
You can find many more overrides option in the official documentation in the Kafka broker listeners chapter.
Prometheus alerts
Other than providing the server component for scraping metrics from your applications, the Prometheus project also provides an alerting system through the alert manager component. It is possible to declare alerting rules on the Prometheus server in order to be notified about specific conditions in the metrics. When an alert condition is evaluated as true, Prometheus sends alert data to the alert manager which then sends notifications out. Notifications can be sent via methods such as email, Slack, PagerDuty and HipChat.
The new Strimzi release provides the Kubernetes/OpenShift resources for deploying the alert manager as well as a few examples of alerting rules for Kafka and Zookeeper metrics and related notifications via Slack.
Here’s an example snippet with an alert related to the under replicated partitions metric.
- alert: UnderReplicatedPartitions
expr: kafka_server_replicamanager_underreplicatedpartitions > 0
for: 10s
labels:
severity: warning
annotations:
summary: 'Kafka under replicated partitions'
description: 'There are {{ $value }} under replicated partitions on {{ $labels.kubernetes_pod_name }}'
… and many more
Other features were included in this new release, the most important ones:
- Support for the Cluster Operator to watch all namespaces for the supported resources.
- Allow users to configure the default ImagePullPolicy related to the containers images download.
- Operator Lifecycle Manager integration which allowed to have Strimzi as part of the OperatorHub.io. You can get more information on this post
Of course, bug fixes are there as well!
Conclusion
This release represents another milestone for this open source project. You can refer to the release change log to get more information.
What are you waiting for? Engage with the community and help us to improve the Strimzi project for running your Kafka cluster on Kubernetes/OpenShift!
Share this:
