Cruise Control reduces the time and effort involved in running an efficient and balanced Kafka cluster.
A typical cluster can become unevenly loaded over time.
Partitions that handle large amounts of message traffic might be unevenly distributed across the available brokers.
To rebalance the cluster, administrators must monitor the load on brokers and manually reassign busy partitions to brokers with spare capacity.
Cruise Control automates the cluster rebalancing process.
It constructs a workload model of resource utilization for the cluster—based on CPU, disk, and network load—and generates optimization proposals (that you can approve or reject) for more balanced partition assignments. A set of configurable optimization goals is used to calculate these proposals.
When you approve an optimization proposal, Cruise Control applies it to your Kafka cluster.
When the cluster rebalancing operation is complete, the broker pods are used more effectively and the Kafka cluster is more evenly balanced.
To rebalance a Kafka cluster, Cruise Control uses optimization goals to generate optimization proposals, which you can approve or reject.
Optimization goals are constraints on workload redistribution and resource utilization across a Kafka cluster.
With a few exceptions, Strimzi supports all the optimization goals developed in the Cruise Control project.
These are as follows, in descending priority order:
-
Rack-awareness
-
Replica capacity
-
Capacity: Disk capacity, Network inbound capacity, Network outbound capacity
-
Replica distribution
-
Potential network output
-
Resource distribution: Disk utilization distribution, Network inbound utilization distribution, Network outbound utilization distribution
-
Leader bytes-in rate distribution
-
Topic replica distribution
-
Leader replica distribution
-
Preferred leader election
For more information on each optimization goal, see Goals in the Cruise Control Wiki.
Note
|
CPU goals, intra-broker disk goals, "Write your own" goals, and Kafka assigner goals are not yet supported.
|
As described in the following sections, you can customize the supported optimization goals by reordering them in terms of priority, and disabling goals to exclude from optimization proposal calculations.
Goals configuration in Strimzi
You configure optimization goals in the Kafka
and KafkaRebalance
custom resources. Cruise Control has configurations for hard optimization goals that must be satisfied, as well as master, default, and user-provided optimization goals.
The following sections describe each goal configuration in more detail.
Hard goals and soft goals
Hard goals are goals that must be satisfied in optimization proposals.
Goals that are not hard goals are known as soft goals, and might not be satisfied by optimization proposals.
You can think of soft goals as best effort goals: they do not need to be satisfied in optimization proposals, but are included in optimization calculations.
An optimization proposal that violates one or more soft goals, but satisfies all hard goals, is valid.
Cruise control will calculate optimization proposals that satisfy all the hard goals and as many soft goals as possible (in their priority order).
An optimization proposal that does not satisfy all the hard goals is rejected by Cruise Control and not sent to the user for approval.
Note
|
For example, you might have a soft goal to distribute a topic’s replicas evenly across the cluster (the topic replica distribution goal).
Cruise Control will ignore this goal if doing so enables all the configured hard goals to be met.
|
RackAwareGoal; ReplicaCapacityGoal; DiskCapacityGoal; NetworkInboundCapacityGoal; NetworkOutboundCapacityGoal; CpuCapacityGoal
Hard goals are controlled in the Cruise Control deployment configuration, by editing the hard.goals
property in Kafka.spec.cruiseControl.config
.
-
To inherit the six preset hard goals from Cruise Control, do not specify the hard.goals
property in Kafka.spec.cruiseControl.config
-
To change the preset hard goals, specify the desired goals in the hard.goals
configuration option.
Increasing the number of hard goals will reduce the likelihood of Cruise Control generating valid optimization proposals.
Note
|
If skipHardGoalCheck: true is specified in the KafkaRebalance custom resource, Cruise Control does not check that the list of user-provided optimization goals (goals ) contains all the configured hard goals (hard.goals ). Therefore, if some, but not all, of the user-provided optimization goals are in the hard.goals list, Cruise Control will still treat them as hard goals even if skipHardGoalCheck: true is specified.
|
Master optimization goals
The master optimization goals are available to all users.
Goals that are not listed in the master optimization goals are not available to use for Cruise Control operations.
Unless you change the Cruise Control deployment configuration, Strimzi will inherit the following master optimization goals from Cruise Control, in descending priority order:
RackAwareGoal; ReplicaCapacityGoal; DiskCapacityGoal; NetworkInboundCapacityGoal; NetworkOutboundCapacityGoal; CpuCapacityGoal; ReplicaDistributionGoal; PotentialNwOutGoal; DiskUsageDistributionGoal; NetworkInboundUsageDistributionGoal; NetworkOutboundUsageDistributionGoal; CpuUsageDistributionGoal; TopicReplicaDistributionGoal; LeaderReplicaDistributionGoal; LeaderBytesInDistributionGoal; PreferredLeaderElectionGoal
To reduce complexity, we recommend that you use the inherited master optimization goals, unless you need to completely exclude one or more goals from use in KafkaRebalance
resources. The priority order of the master optimization goals can be modified, if desired, in the default optimization goals.
Master optimization goals are controlled, if needed, in the Cruise Control deployment configuration: Kafka.spec.cruiseControl.config.goals
-
To accept the inherited master optimization goals, do not specify the goals
property in Kafka.spec.cruiseControl.config
.
-
If you need to modify the inherited master optimization goals, specify a list of goals in the goals
configuration option.
Default optimization goals
Cruise Control uses the default optimization goals to generate the cached optimization proposal.
For more information about the cached optimization proposal, see Optimization proposals overview.
Unless you change the Cruise Control deployment configuration, the default optimization goals are the same as the master optimization goals.
-
To use the master optimization goals as the default goals, do not specify the default.goals
property in Kafka.spec.cruiseControl.config
.
-
To modify the default optimization goals, edit the default.goals
property in Kafka.spec.cruiseControl.config
.
You must use a subset of the master optimization goals.
Example Kafka
configuration for default optimization goals
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
name: my-cluster
spec:
kafka:
# ...
zookeeper:
# ...
entityOperator:
topicOperator: {}
userOperator: {}
cruiseControl: {}
capacity:
networkIn: 10000KB/s
networkOut: 10000KB/s
config:
default.goals: >
com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,
com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,
com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal
# ...
If no default optimization goals are specified, the cached proposal is generated using the master optimization goals.
User-provided optimization goals
User-provided optimization goals narrow down the default goals.
You can set them, if required, in the KafkaRebalance
custom resource for a particular optimization proposal: KafkaRebalance.spec.goals
They are useful for generating an optimization proposal that addresses a particular scenario.
For example, you might want to optimize leader replica distribution across the Kafka cluster without considering goals for disk capacity or disk utilization.
So, you create a KafkaRebalance
custom resource containing a user-provided goal for leader replica distribution only.
User-provided optimization goals must:
To ignore the configured hard goals in an optimization proposal, add the skipHardGoalCheck: true
option to the KafkaRebalance
custom resource.
An optimization proposal is a summary of proposed changes that would produce a more balanced Kafka cluster, with partition workloads distributed more evenly among the brokers.
Each optimization proposal is based on the set of optimization goals that were used to generate it.
Use the summary (shown in the status
of the KafkaRebalance
resource) to decide whether to:
-
Approve the optimization proposal. This instructs Cruise Control to apply the proposal to the Kafka cluster and start a cluster rebalance operation.
-
Reject the optimization proposal. You can change the optimization goals and then generate another proposal.
Note
|
All optimization proposals are dry runs: you cannot approve a cluster rebalance without first generating an optimization proposal.
There is no limit to the number of optimization proposals that can be generated.
|
Cached optimization proposal
Cruise Control maintains a cached optimization proposal based on the configured default optimization goals.
Generated from the workload model, the cached optimization proposal is updated every 15 minutes to reflect the current state of the Kafka cluster.
If you generate an optimization proposal using the default optimization goals, Cruise Control returns the most recent cached proposal.
To change the cached optimization proposal refresh interval, edit the proposal.expiration.ms
setting in the Cruise Control deployment configuration.
Consider a shorter interval for fast changing clusters, although this increases the load on the Cruise Control server.
Contents of optimization proposals
The following table explains the properties contained in an optimization proposal:
JSON property |
Description |
numIntraBrokerReplicaMovements
|
The total number of partition replicas that will be transferred between the disks of the cluster’s brokers.
Performance impact during rebalance operation: Relatively high, but lower than numReplicaMovements . |
excludedBrokersForLeadership
|
Not yet supported. An empty list is returned. |
numReplicaMovements
|
The number of partition replicas that will be moved between separate brokers.
Performance impact during rebalance operation: Relatively high. |
onDemandBalancednessScoreBefore, onDemandBalancednessScoreAfter
|
A measurement of the overall balancedness of a Kafka Cluster, before and after the optimization proposal was generated.
The calculation is 100 minus the sum of the BalancednessScore of each violated soft goal. Cruise Control assigns a BalancednessScore to every optimization goal based on several factors, including priority (the goal’s position in the list of default.goals or user-provided goals).
The Before score is based on the current configuration of the Kafka cluster.
The After score is based on the generated optimization proposal. |
intraBrokerDataToMoveMB
|
The sum of the size of each partition replica that will be moved between disks on the same broker (see also numIntraBrokerReplicaMovements ).
Performance impact during rebalance operation: Variable. The larger the number, the longer the cluster rebalance will take to complete. Moving a large amount of data between disks on the same broker has less impact than between separate brokers (see dataToMoveMB ). |
recentWindows
|
The number of metrics windows upon which the optimization proposal is based. |
dataToMoveMB
|
The sum of the size of each partition replica that will be moved to a separate broker (see also numReplicaMovements ).
Performance impact during rebalance operation: Variable. The larger the number, the longer the cluster rebalance will take to complete. |
monitoredPartitionsPercentage
|
The percentage of partitions in the Kafka cluster covered by the optimization proposal. Affected by the number of excludedTopics . |
excludedTopics
|
Not yet supported. An empty list is returned. |
numLeaderMovements
|
The number of partitions whose leaders will be switched to different replicas. This involves a change to ZooKeeper configuration.
Performance impact during rebalance operation: Relatively low. |
excludedBrokersForReplicaMove
|
Not yet supported. An empty list is returned. |
Cruise Control is configured using the cruiseControl
property in the Kafka
resource.
Once configured, you can deploy a Cruise Control instance to your Strimzi cluster by creating or updating the Kafka
resource.
You can configure cruiseControl
properties as part of a deployment or redeployment of a Kafka cluster.
Deploy one instance of Cruise Control per Kafka cluster.
This procedure uses the following example file provided with Strimzi:
This creates a strimzi.cruisecontrol.metrics
topic in your Kafka cluster, which collects information from the Cruise Control Metrics Reporter for use in generating rebalance proposals.
You do not need to interact with this topic after Cruise Control is deployed.
Procedure
-
Create the Cruise Control metrics topic in your Kafka cluster:
kubectl apply -f examples/cruise-control/cruise-control-topic.yaml
-
Edit the cruiseControl
property of the Kafka
resource.
The properties you can configure are shown in this example configuration:
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
name: my-cluster
spec:
# ...
cruiseControl:
capacity: (1)
networkIn: 10000KB/s
networkOut: 10000KB/s
# ...
config: (2)
default.goals: >
com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,
com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal
# ...
goals: {}
cpu.balance.threshold: 1.1
metadata.max.age.ms: 300000
send.buffer.bytes: 131072
# ...
resources: (3)
requests:
cpu: 200m
memory: 64Mi
limits:
cpu: 500m
memory: 128Mi
logging: (4)
type: inline
loggers:
cruisecontrol.root.logger: "INFO"
template: (5)
pod:
metadata:
labels:
label1: value1
securityContext:
runAsUser: 1000001
fsGroup: 0
terminationGracePeriodSeconds: 120
readinessProbe: (6)
initialDelaySeconds: 15
timeoutSeconds: 5
livenessProbe: (7)
initialDelaySeconds: 15
timeoutSeconds: 5
# ...
-
Create or update the resource:
kubectl apply -f kafka.yaml
-
Verify that Cruise Control was successfully deployed:
kubectl get deployments -l app.kubernetes.io/name=strimzi
The config
property in Kafka.spec.cruiseControl
contains configuration options as keys with values as one of the following JSON types:
Note
|
Strings that look like JSON or YAML will need to be explicitly quoted.
|
You can specify and configure all the options listed in the "Configurations" section of the Cruise Control documentation, apart from those managed directly by Strimzi.
Specifically, you cannot modify configuration options with keys equal to or starting with one of the following strings:
-
bootstrap.servers
-
zookeeper.
-
ssl.
-
security.
-
failed.brokers.zk.path
-
webserver.http.port
-
webserver.http.address
-
webserver.api.urlprefix
-
metric.reporter.sampler.bootstrap.servers
-
metric.reporter.topic
-
metric.reporter.topic.pattern
-
partition.metric.sample.store.topic
-
broker.metric.sample.store.topic
-
capacity.config.file
-
skip.sample.store.topic.rack.awareness.check
-
cruise.control.metrics.topic
-
sasl.
If restricted options are specified, they are ignored and a warning message is printed to the Cluster Operator log file.
All supported options are passed to Cruise Control.
An example Cruise Control configuration
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
name: my-cluster
spec:
# ...
cruiseControl:
# ...
config:
default.goals: >
com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,
com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal
cpu.balance.threshold: 1.1
metadata.max.age.ms: 300000
send.buffer.bytes: 131072
# ...
Cruise Control uses capacity limits to determine if optimization goals for broker resources are being broken.
An optimization will fail if a goal is broken, preventing the optimization from being used to generate an optimization proposal for balanced partition assignments.
For example, an optimization that would cause a broker to exceed its CPU capacity would not be used if the CpuCapacityGoal
is set as a hard goal.
You specify capacity limits for broker resources in the brokerCapacity
property in Kafka.spec.cruiseControl
.
Capacity limits can be set for the following broker resources in the described units:
-
disk
- Disk storage in bytes
-
cpuUtilization
- CPU utilization as a percent (0-100)
-
inboundNetwork
- Inbound network throughput in bytes per second
-
outboundNetwork
- Outbound network throughput in bytes per second
Because Strimzi Kafka brokers are homogeneous, Cruise Control applies the same capacity limits to every broker it is monitoring.
An example Cruise Control brokerCapacity configuration
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
name: my-cluster
spec:
# ...
cruiseControl:
# ...
brokerCapacity:
disk: 100G
cpuUtilization: 100
inboundNetwork: 10000KB/s
outboundNetwork: 10000KB/s
# ...
Cruise Control requires the following three topics to be created in order to function properly:
-
strimzi.cruisecontrol.partitionmetricsamples
-
strimzi.cruisecontrol.modeltrainingsamples
-
strimzi.cruisecontrol.metrics
Cruise Control will create the strimzi.cruisecontrol.partitionmetricsamples
and strimzi.cruisecontrol.modeltrainingsamples
topics after the Cruise Control metric reporters create the strimzi.cruisecontrol.metrics
topic.
However, if automatic topic creation is disabled in Kafka with a configuration of auto.create.topics.enable: false
in spec.kafka.config
when starting a new Kafka cluster, the Cruise Control metric reporters will be unable to create the strimzi.cruisecontrol.metrics
topic.
In this situation, the strimzi.cruisecontrol.metrics
topic will need to be created manually.
To manually create the strimzi.cruisecontrol.metrics
Cruise Control topic in your Kubernetes cluster:
kubectl apply -f examples/cruise-control/cruise-control-topic.yaml
Since log compaction may remove records needed by Cruise Control, all topics created by Cruise Control must be configured with cleanup.policy=delete
to disable log compaction.
Cruise Control will automatically disable log compaction for the strimzi.cruisecontrol.partitionmetricsamples
and strimzi.cruisecontrol.modeltrainingsamples
topics.
The Cruise Control metric reporters will attempt to disable log compaction for the strimzi.cruisecontrol.metrics
topic but will fail when being started with a new Kafka cluster.
This will only become a problem when log compaction is enabled in Kafka with the setting log.cleanup.policy=compact
in the spec.kafka.config
.
In this situation, log compaction will be enabled for strimzi.cruisecontrol.metrics
topic and will need to be overridden with a cleanup.policy=delete
in the strimzi.cruisecontrol.metrics
KafkaTopic.
Here we see an example of log compaction being disabled in a Cruise Control KafkaTopic.
apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaTopic
metadata:
name: strimzi.cruisecontrol.metrics
spec:
partitions: 1
replicas: 1
config:
cleanup.policy: delete
Cruise Control has its own configurable logger:
Cruise Control uses the Apache log4j
logger implementation.
Use the logging
property to configure loggers and logger levels.
You can set the log levels by specifying the logger and level directly (inline) or use a custom (external) ConfigMap.
If a ConfigMap is used, you set logging.name
property to the name of the ConfigMap containing the external logging configuration. Inside the ConfigMap, the logging configuration is described using log4j.properties
.
Here we see examples of inline
and external
logging.
Inline logging
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
# ...
spec:
cruiseControl:
# ...
logging:
type: inline
loggers:
cruisecontrol.root.logger: "INFO"
# ...
External logging
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
# ...
spec:
cruiseControl:
# ...
logging:
type: external
name: customConfigMap
# ...
If an issue occurs in the process of creating a KafkaRebalance
resource, or when interacting with Cruise Control, the error is reported in the resource status, along with details of how to fix it.
The resource also moves to the NotReady
state.
To continue with the cluster rebalance operation, you must fix the problem in the KafkaRebalance
resource itself.
Problems can include the following:
When you believe that you have fixed the issue, you need to add the refresh
annotation to the resource.
During a “refresh”, a new optimization proposal is requested from the Cruise Control server.
Procedure
-
Get information about the error from the KafkaRebalance
status and resolve the issue if possible.
-
Annotate the KafkaRebalance
resource in Kubernetes:
kubectl annotate kafkarebalance cluster-rebalance-name strimzi.io/rebalance=refresh
-
Check the status of the KafkaRebalance
resource:
kubectl describe kafkarebalance rebalance-cr-name
-
Wait until the status changes to PendingProposal
, or directly to ProposalReady
.