At LittleHorse, we use the Gateway API to allow external traffic into our Kubernetes clusters. We will soon need to allow external clients to access our Kafka clusters (managed by Strimzi, of course!) from outside of our Kubernetes clusters. We have so far been pleased with the performance and simplicity of Envoy Gateway as a Gateway Controller, which motivated this investigation into using Envoy Gateway to access our Kafka clusters.
Background
The Gateway API in Kubernetes aims to replace the Ingress
resource as the de facto standard for allowing external traffic to reach workloads running on Kubernetes. It addresses many shortcomings of the Ingress
resource, including poor support for non-HTTP 1.0 traffic.
The Gateway API has many implementations. In this blog we will use Envoy Gateway as our Gateway Controller. We chose Envoy Gateway for production at LittleHorse due to its simple deployment model, Envoy’s maturity and performance, and our extensive experience with Envoy.
Accessing Kafka
Previously, Jakub Scholz blogged about how Strimzi allows you to access Kafka from outside the Kubernetes cluster using NodePort
services, OpenShift Route
s, LoadBalancer
Services, and Ingress
resources.
As Jakub noted, accessing Kafka is difficult for two reasons:
- Kafka Clients need to be able to access specific brokers individually, so simply scattering the requests across the Kafka Cluster using a load balancer would yield incorrect results.
- The Kafka protocol is not based on HTTP, which means that you need a few clever hacks to get it to work with plain
Ingress
.
The Gateway API
The Gateway API is a much more flexible and extensible solution for north-south traffic than Ingress
. The entirety of the Gateway API is beyond the scope of this post, but there are two resources in particular that will be of interest to us:
TCPRoute
resources, which proxy unencrypted TCP traffic.TLSRoute
resources, which control encrypted TCP traffic.
The HTTPRoute
and GRPCRoute
resources will not work with Kafka because Kafka does not speak an HTTP or gRPC-based protocol.
NOTE: The HTTPRoute
resource has reached GA in the Gateway API; however, the TLSRoute
is still in beta. Some advise not using it, but it should be fine as long as you check release notes and test before upgrading!
This post will focus on the TLSRoute
. In particular, we will use passthrough TLS in which the TLS connections are terminated not at the Gateway Controller but rather at the Kafka brokers.
Putting It Into Practice
The rest of this blog post will walk through how to use TLSRoute
s to access a Strimzi-managed Kafka cluster from outside of your Kubernetes cluster. We will use a KIND cluster, which allows us to run a Kubernetes cluster in docker containers on our local machine, and we will use Envoy Gateway as our implementation of the Gateway API.
If you would like to follow along without copying and pasting yaml files, you can clone a GitHub repo that I made which has all of the following code: https://github.com/coltmcnealy-lh/strimzi-gateway-api.
Local Environment Overview
This example will utilize a few small hacks to make it possible to do local development with your KIND cluster. We want to access Kafka using a TLS-encrypted connection, which means that:
- We’ll have to create a certificate with some hostname for the Kafka brokers.
- We’ll need to be able to somehow redirect traffic from that hostname into the KIND cluster before the
TLSRoute
Gateway Controller is even able to route the traffic to Kafka.
KIND is very flexible and has some options to do this. What we will do is:
- Create a self-signed certificate for the url
*.strimzi.gateway.api.test
and configure our Kafka clients to trust it. - Use the
/etc/hosts
file to map some url’s ending in.strimzi.gateway.api.test
to your localhost. - Map port
9092
on your local terminal to port30992
on the KIND node (which is just a docker container running on your laptop). - Deploy the Envoy Gateway pods with a
NodePort
service mapping port30992
on the K8s node to port9092
on the Envoy Gateway Pod.
Together, the four steps above will make it possible to send traffic to your KIND cluster as if it were running in a public network.
Lastly, once we have traffic successfully routed to the Envoy Proxy (Gateway Controller) pod, we will use a TLSRoute
resource to ensure that traffic reaches our Kafka brokers. The entire networking setup can be visualized as follows:
- From your local terminal, you make a request to
boostrap.strimzi.gateway.api.test:9092
. - The
/etc/hosts
file re-routes it tolocalhost
. - The KIND cluster has a node running as a docker container with your host port
9092
mapped to container port30992
. - There is a
NodePort
Service routing traffic from port30992
on the Kubernetes Nodes to the Envoy Gateway pod. - We use
TLSRoute
resources to configure Envoy to send traffic to the Kafka brokers.
KIND Cluster Setup
First, let’s create the KIND cluster using the following kind-config.yaml
file:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- kubeadmConfigPatches:
- |
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
node-labels: "ingress-ready=true"
extraPortMappings:
- containerPort: 30992
hostPort: 9092
protocol: TCP
If you inspect the kind-config.yaml
file, you will notice the port mapping of hostPort: 9092
being mapped to containerPort: 30992
. That means that the port 9092 on your own laptop will be forwarded by docker to port 30992 on the Kubernetes Node (which, in KIND, is just a docker container running on your laptop). We will use this fact when installing Envoy Gateway.
If you’re following along in my GitHub repo, you can run the following:
kind create cluster --name strimzi-gw-api --config kind-config.yaml
# Use a namespace called "strimzi"
kubectl create ns strimzi
kubectl config set-context --current --namespace=strimzi
Next, let’s set up Envoy Gateway. First, we will use helm
to install it. Note that a lot of Envoy Gateway tooling (including egctl
, the Envoy Gateway CLI) expects it to be installed in the envoy-gateway-system
namespace, so we will do that. Strimzi is flexible enough to go anywhere.
helm upgrade --install envoygateway oci://docker.io/envoyproxy/gateway-helm \
--version v1.0.1 \
--namespace envoy-gateway-system \
--create-namespace
Once the installation process is complete, we need to deploy a Gateway
with a specific GatewayClass
that will listen on the correct NodePort
s. We’ll refer to this Gateway
later when we create TLSRoute
s to access our Kafka cluster.
Before creating the Gateway
, we must create and configure the GatewayClass
. A GatewayClass
is just like an IngressClass
: it defines a type of Gateway which can be reconciled by a Gateway Controller. Envoy Gateway also has an additional CRD called EnvoyProxy
which can be attached to a GatewayClass
to tell Envoy Gateway how to reconcile gateways of that class.
If you recall from earlier, we want to create a GatewayClass
that is configured to run Envoy Proxy pods with a NodePort
service type, mapping port 30992
on the K8s Node to port 9092
on the Envoy Proxy pods. You can do that as follows:
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: my-gateway-class
namespace: envoy-gateway-system
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: my-proxy-config
namespace: envoy-gateway-system
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: my-proxy-config
namespace: envoy-gateway-system
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
replicas: 1
envoyService:
type: NodePort
patch:
value:
spec:
ports:
# Port 9092 on your laptop gets forwarded to NodePort 30992 on the KIND cluster.
- name: kafka-port
nodePort: 30992
port: 9092
protocol: TCP
targetPort: 9092
---
If you’re following along in my GitHub:
kubectl apply -f gateway-class.yaml
Now that we have a GatewayClass
which is properly configured by an Envoy Gateway EnvoyProxy
resource, we can create our Gateway
. As discussed earlier in the setup overview, we want the Envoy Proxy pods to listen on port 9092
. That listener will be configured to have Passthrough TLS.
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: my-gateway
# Envoy Gateway can control gateways in all namespaces. Depending on your
# cluster permissions model, a Gateway can be in the envoy-gateway-system namespace
# or in the same namespace as your applications.
namespace: strimzi
spec:
gatewayClassName: my-gateway-class
listeners:
- name: kafka-listener
protocol: TLS
port: 9092
tls:
mode: Passthrough
In the github repo:
kubectl apply -f gateway.yaml
This will create a Gateway
resource, and the Envoy Gateway controller will deploy an Envoy pod in the envoy-gateway-system
namespace to handle traffic for that gateway class. Once that is done, you should see some pods in the envoy-gateway-system
namespace, like the following:
-> kubectl get pods --namespace envoy-gateway-system
NAME READY STATUS RESTARTS AGE
envoy-strimzi-my-gateway-1c7c06f0-5446c7ff7b-vpd6m 1/2 Running 0 22s
envoy-gateway-8595cc9fbc-2bjn5 1/1 Running 0 96s
The second pod is the Envoy Gateway controller, which reconciles all Gateway API-related resources. The first Pod
was created by the controller to route all traffic for the my-gateway
Gateway
which we created in the strimzi
namespace.
Next, the most exciting part about the setup process is installing Strimzi. You can do it as follows:
helm upgrade --install strimzi oci://quay.io/strimzi-helm/strimzi-kafka-operator \
--version 0.42.0 \
--namespace strimzi
Since the TLSRoute
resource uses passthrough TLS, in which encryption is terminated at the Kafka broker pods, we’ll need a TLS certificate to mount on the Kafka brokers. While we could use openssl
, in this example we’ll use another operator, Cert Manager, to create TLS certificates for us using the Certificate
resource.
You can install Cert Manager as follows:
helm upgrade --install cert-manager jetstack/cert-manager \
--namespace strimzi \
--version v1.15.1 \
--set installCRDs=true
Next, create a self-signed Issuer
and a Certificate
:
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: my-certificate
namespace: strimzi
spec:
secretName: my-certificate
subject:
organizations:
- my-org
privateKey:
algorithm: RSA
encoding: PKCS8
size: 2048
usages:
- server auth
- client auth
dnsNames:
- '*.strimzi.gateway.api.test'
issuerRef:
name: my-issuer
kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: my-issuer
namespace: strimzi
spec:
selfSigned: {}
In the GitHub repo, you can follow along via:
kubectl apply -f certificate.yaml
You should be able to see a Secret
named my-certificate
in the strimzi
namespace.
The last piece of setup is to configure your /etc/hosts
file so that you can access Kafka from outside of the cluster. This is one of two hacks that we will use to get this example to work on your local KIND box—in real life, you would probably use real DNS records to point to your Kubernetes Cluster. In this case, we want to make *.strimzi.gateway.api.test
point to localhost
so that it ends up hitting the KIND node (which is just a docker container running on localhost
).
Add the following to /etc/hosts
:
# For Strimzi Gateway API
127.0.0.1 bootstrap.strimzi.gateway.api.test
127.0.0.1 broker-10.strimzi.gateway.api.test
127.0.0.1 broker-11.strimzi.gateway.api.test
127.0.0.1 broker-12.strimzi.gateway.api.test
We will:
- Make it so that any traffic sent to those endpoints (on port
9092
) ends up at the Envoy Gateway pod(s). - Configure our Kafka cluser to advertise the above endpoints.
- Create
TLSRoute
s that route traffic from the Envoy Gateway pods to the appropriate Kafka brokers using the Server Name Indication protocol.
Deploying the Kafka
Cluster
Let’s create a Kafka cluster in KRaft mode. Our cluster will have 1 Controller and 3 Brokers. This means we’re going to need a single Kafka
resource and two KafkaNodePool
s.
Our Kafka
cluster will have one listener on it, on port 9092
:
# ...
spec:
kafka:
listeners:
- port: 9092
# ...
We’ll need to use the cluster-ip
listener type so that Strimzi creates a Service
of type: ClusterIP
for each broker. This enables us to use TLSRoute
s to send traffic to the proper broker on the backend.
# ...
- port: 9092
type: cluster-ip
# ...
Additionally, since we’re exposing our Kafka cluster to the internet, we’ll use scram-sha-512
authentication to secure the listener:
# ...
authentication:
type: scram-sha-512
# ...
Rather than have Strimzi create the certificates for us, we want to use the my-certificate
secret created by Cert Manager. Most importantly, we need to configure the advertised listeners on the brokers so that they advertise the endpoints that we set above.
NOTE: we will configure the broker KafkaNodePool
to start with node id 10
so that we guarantee that we expose only the brokers and not the controllers.
configuration:
brokerCertChainAndKey:
certificate: tls.crt
key: tls.key
secretName: my-certificate
brokers:
- advertisedHost: broker-10.strimzi.gateway.api.test
advertisedPort: 9092
broker: 10
- advertisedHost: broker-11.strimzi.gateway.api.test
advertisedPort: 9092
broker: 11
- advertisedHost: broker-12.strimzi.gateway.api.test
advertisedPort: 9092
broker: 12
createBootstrapService: true
Putting it together, the Kafka
resource looks like the following:
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: gateway-api-test
namespace: strimzi
annotations:
strimzi.io/kraft: enabled
strimzi.io/node-pools: enabled
spec:
entityOperator:
# We'll create a kafka topic and user so we need these operators.
topicOperator: {}
userOperator: {}
kafka:
authorization:
type: simple
listeners:
- authentication:
type: scram-sha-512
configuration:
brokerCertChainAndKey:
certificate: tls.crt
key: tls.key
secretName: my-certificate
brokers:
- advertisedHost: broker-10.strimzi.gateway.api.test
advertisedPort: 9092
broker: 10
- advertisedHost: broker-11.strimzi.gateway.api.test
advertisedPort: 9092
broker: 11
- advertisedHost: broker-12.strimzi.gateway.api.test
advertisedPort: 9092
broker: 12
createBootstrapService: true
name: obiwan
port: 9092
tls: true
type: cluster-ip
version: 3.7.1
If you’re following along on GitHub, it’s just:
kubectl apply -f kafka.yaml
Next, we need to create the KafkaNodePool
s for the controllers and the brokers. We will ensure that the controller node id’s start at 0
and the brokers start at 10
. Our cluster will only have one controller, but in production it’s recommended to use 3. Note that before KIP-853 is completed in Apache Kafka and supported in Strimzi, it is not possible to change the number of controllers in your cluster once it’s deployed. Hopefully, that will be addressed in the upcoming Kafka 3.9.0
release and soon after in Strimzi.
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
annotations:
strimzi.io/next-node-ids: '[10-100]'
labels:
strimzi.io/cluster: gateway-api-test
name: broker
namespace: strimzi
spec:
replicas: 3
roles:
- broker
storage:
class: standard
size: 10G
type: persistent-claim
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
annotations:
strimzi.io/next-node-ids: '[0-9]'
labels:
strimzi.io/cluster: gateway-api-test
name: controller
namespace: strimzi
spec:
replicas: 1
roles:
- controller
storage:
class: standard
size: 10G
type: persistent-claim
In the github repository, you can create the above resources via the following command:
kubectl apply -f kafka-node-pools.yaml
At this point, you should see the Kafka pods up and running.
NAME READY STATUS RESTARTS AGE
cert-manager-84489bc478-7cxds 1/1 Running 0 7m22s
cert-manager-cainjector-7477d56b47-ct8dv 1/1 Running 0 7m22s
cert-manager-webhook-6d5cb854fc-2rx9p 1/1 Running 0 7m22s
gateway-api-test-broker-10 1/1 Running 0 82s
gateway-api-test-broker-11 1/1 Running 0 82s
gateway-api-test-broker-12 1/1 Running 0 82s
gateway-api-test-controller-0 1/1 Running 0 82s
gateway-api-test-entity-operator-6657fbc775-w4b65 2/2 Running 0 44s
strimzi-cluster-operator-6948497896-s7q46 1/1 Running 0 2m23s
Creating TLSRoute
s
Next, we will need to create four TLSRoute
s:
- A
bootstrap
one that points to the bootstrap service. - A
TLSRoute
for each broker that is deployed.
The TLSRoute
definitions look like this:
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSRoute
metadata:
name: gateway-api-test-broker-10
namespace: strimzi
spec:
hostnames:
- broker-10.strimzi.gateway.api.test
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: my-gateway
namespace: strimzi
sectionName: kafka-listener
rules:
- backendRefs:
- group: ""
kind: Service
name: gateway-api-test-broker-obiwan-10
port: 9092
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSRoute
metadata:
name: gateway-api-test-broker-11
namespace: strimzi
spec:
hostnames:
- broker-11.strimzi.gateway.api.test
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: my-gateway
namespace: strimzi
sectionName: kafka-listener
rules:
- backendRefs:
- group: ""
kind: Service
name: gateway-api-test-broker-obiwan-11
port: 9092
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSRoute
metadata:
name: gateway-api-test-broker-12
namespace: strimzi
spec:
hostnames:
- broker-12.strimzi.gateway.api.test
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: my-gateway
namespace: strimzi
sectionName: kafka-listener
rules:
- backendRefs:
- group: ""
kind: Service
name: gateway-api-test-broker-obiwan-12
port: 9092
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSRoute
metadata:
name: gateway-api-test-bootstrap
namespace: strimzi
spec:
hostnames:
- bootstrap.strimzi.gateway.api.test
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: my-gateway
namespace: strimzi
sectionName: kafka-listener
rules:
- backendRefs:
- group: ""
kind: Service
name: gateway-api-test-kafka-obiwan-bootstrap
port: 9092
To follow along in the github repo, you can run:
kubectl apply -f tls-routes.yaml
Creating a Kafka Client Config
In order to access Kafka, we will create a KafkaUser
that will create credentials as a Kubernetes Secret
for us to access the secured Kafka cluster. We’ll also create a KafkaTopic
to play with in the next section.
First, create a KafkaUser
and KafkaTopic
:
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
name: obiwan
namespace: strimzi
labels:
strimzi.io/cluster: gateway-api-test
spec:
authentication:
type: scram-sha-512
authorization:
type: simple
acls:
- resource:
type: topic
name: "*"
patternType: literal
operations:
- 'All'
host: "*"
- resource:
type: group
name: "*"
patternType: literal
operations:
- 'All'
host: "*"
- resource:
type: cluster
operations:
- 'All'
- resource:
type: transactionalId
name: "*"
patternType: literal
operations:
- 'All'
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
name: my-topic
namespace: strimzi
labels:
strimzi.io/cluster: gateway-api-test
spec:
partitions: 12
replicas: 3
Once again, in github this can be done via the user-and-topic.yaml
file:
kubectl apply -f user-and-topic.yaml
At this point, you should be able to see a KafkaTopic
and KafkaUser
resource created in your Kubernetes cluster:
-> kubectl get kafkauser
NAME CLUSTER AUTHENTICATION AUTHORIZATION READY
obiwan gateway-api-test scram-sha-512 simple True
-> kubectl get kafkatopic
NAME CLUSTER PARTITIONS REPLICATION FACTOR READY
my-topic gateway-api-test 12 3 True
You should now see a Secret
named obiwan
. If you inspect it, you’ll see a sasl.jaas.config
field which contains a base64-encoded String that can be passed for the Kafka sasl.jaas.config
configuration property.
To access Kafka, we need to create a client configuration property. We will need to do the following:
- Download the
sasl.jaas.config
from theSecret
created for theKafkaUser
. - Download the CA Public Cert created by Cert Manager.
- Convert the CA Cert from the previous step into the JKS format so that we can use it in our Kafka config.
The following script does all of that and writes a Kafka config file to /tmp/kafka-client-config.properties
.
#!/bin/bash
kubectl get secret my-certificate -o json | jq '.data."ca.crt"' | tr -d '"' | base64 --decode > /tmp/ca.crt
keytool -importcert -alias ca -file /tmp/ca.crt -keystore /tmp/strimzi-kafka-truststore.jks -storepass kenobi
JAAS_CONFIG=$(kubectl get secret obiwan -o json | jq '.data."sasl.jaas.config"' | tr -d '"' | base64 --decode)
cat <<EOF > /tmp/kafka-client-config.properties
bootstrap.servers=bootstrap.strimzi.gateway.api.test:9092
sasl.jaas.config=$JAAS_CONFIG
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
ssl.truststore.location=/tmp/strimzi-kafka-truststore.jks
ssl.truststore.password=kenobi
EOF
When running the script make sure to type “yes” to add the certificate to the JKS keystore.
./create-kafka-config.sh
cat /tmp/kafka-client-config.properties
Accessing Kafka
The last thing we need to do is use our Kafka cluster! We’ll use the Strimzi Docker images and the kafka-console-{producer,consumer}.sh
scripts.
In one terminal, start a consumer:
docker run -it --rm --network host \
-v /tmp/:/tmp/ quay.io/strimzi/kafka:0.42.0-kafka-3.7.1 \
bin/kafka-console-consumer.sh \
--bootstrap-server bootstrap.strimzi.gateway.api.test:9092 \
--topic my-topic \
--from-beginning \
--consumer.config /tmp/kafka-client-config.properties
And in another, start a producer:
docker run -it --rm --network host \
-v /tmp/:/tmp/ quay.io/strimzi/kafka:0.42.0-kafka-3.7.1 \
bin/kafka-console-producer.sh \
--bootstrap-server bootstrap.strimzi.gateway.api.test:9092 \
--topic my-topic \
--producer.config /tmp/kafka-client-config.properties
Conclusion
Strimzi has native support for Ingress
, OpenShift Route
s, and LoadBalancer
and NodePort
services. This support covers the vast majority of use-cases; however, Strimzi is still flexible enough for you to implement your own custom listeners that use different mechanisms for allowing north-south traffic into your Kafka cluster.
In this example, we have shown you how to securely leverage Envoy Gateway’s implementation of the new Gateway API, which is poised to become the de facto standard for Kubernetes networking over the next few years. For now, extending Strimzi to use the Gateway API is somewhat manual, but who knows…it may become natively-supported within Strimzi some day!