TLS Verification of Upstream Services

From version 3.4.0 Kong Ingress Controller allows you to configure Kong Gateway to run the TLS verification of upstream services. If you have an upstream service that handles TLS, you can configure Kong Gateway to verify the certificate it presents by attaching a CA certificate to a service. This guide shows how to make this happen using the BackendTLSPolicy (when using Gateway API) or Kubernetes Service annotations (when using Ingress API).

Prerequisites: Install Kong Ingress Controller with Gateway API support in your Kubernetes cluster and connect to Kong.

Prerequisites

Install the Gateway APIs

Install the experimental Gateway API CRDs before installing Kong Ingress Controller.

 kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.1.0/experimental-install.yaml

Create a Gateway and GatewayClass instance to use.

echo "
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: kong
  annotations:
    konghq.com/gatewayclass-unmanaged: 'true'

spec:
  controllerName: konghq.com/kic-gateway-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: kong
spec:
  gatewayClassName: kong
  listeners:
  - name: proxy
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
         from: All
" | kubectl apply -f -

The results should look like this:

gatewayclass.gateway.networking.k8s.io/kong created
gateway.gateway.networking.k8s.io/kong created

Install Kong

You can install Kong in your Kubernetes cluster using Helm.

Add the Kong Helm charts:

 helm repo add kong https://charts.konghq.com
 helm repo update

Install Kong Ingress Controller and Kong Gateway with Helm:

 helm install kong kong/ingress -n kong --create-namespace 

Enable the Gateway API Alpha feature gate:

 kubectl set env -n kong deployment/kong-controller CONTROLLER_FEATURE_GATES="GatewayAlpha=true" -c ingress-controller

The results should look like this:

deployment.apps/kong-controller env updated

Test connectivity to Kong

Kubernetes exposes the proxy through a Kubernetes service. Run the following commands to store the load balancer IP address in a variable named PROXY_IP:

Populate $PROXY_IP for future commands:

 export PROXY_IP=$(kubectl get svc --namespace kong kong-gateway-proxy -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
 echo $PROXY_IP

Ensure that you can call the proxy IP:

 curl -i $PROXY_IP

The results should look like this:

 HTTP/1.1 404 Not Found
 Content-Type: application/json; charset=utf-8
 Connection: keep-alive
 Content-Length: 48
 X-Kong-Response-Latency: 0
 Server: kong/3.0.0
  
 {"message":"no Route matched with those values"}

Set up an upstream service with TLS

Before we configure Kong Ingress Controller to verify the certificate of the upstream service, we need to set up an upstream service that handles TLS. We can use the kong/goecho service as an example. It is a simple echo server that can be configured to serve HTTPS on an arbitrary port using a TLS certificate and key pair.

Create a certificate chain

To showcase all the possible configurations, we will create a certificate chain with a root CA, an intermediate CA, and a leaf server certificate. We will use openssl to generate the certificates.

# Create a directory to store the certificates
mkdir certs
cd certs

# Generate Root CA
openssl req -new -newkey rsa:2048 -nodes -keyout root.key -subj "/CN=root" -x509 -days 365 -out root.crt

# Generate Intermediate CA
openssl req -new -newkey rsa:2048 -nodes -keyout inter.key -subj "/CN=inter" -out inter.csr
openssl x509 -req -in inter.csr -CA root.crt -CAkey root.key -CAcreateserial -days 365 -out inter.crt -extfile <(echo "basicConstraints=CA:TRUE")

# Generate Leaf Certificate
openssl req -new -newkey rsa:2048 -nodes -keyout leaf.key -subj "/CN=kong.example" -out leaf.csr
openssl x509 -req -in leaf.csr -CA inter.crt -CAkey inter.key -CAcreateserial -days 365 -out leaf.crt -extfile <(printf "subjectAltName=DNS:kong.example")

# Create a certificate chain
cat leaf.crt inter.crt > chain.crt

# Cleanup intermediate files
rm -f *.csr *.srl
cd ..

Running this script will generate the following files in certs directory:

root.key and root.crt: Root CA key and certificate
inter.key, inter.crt: Intermediate CA key and certificate
leaf.key, leaf.crt: Server key and certificate (valid for kong.example SAN)
chain.crt: Server certificate chain

Deploy the goecho service

We will deploy the kong/goecho service with the generated certificates. We will use the leaf.key and chain.crt files to configure the service to serve HTTPS.

First, deploy the standard kong/goecho service (no HTTPS support).

kubectl apply -f https://docs.jp.konghq.com/assets/kubernetes-ingress-controller/examples/echo-service.yaml

The results should look like this.

service/echo created
deployment.apps/echo created

Now, let’s configure it to serve HTTPS. Let’s create a secret with the server key and the certificate chain (including the intermediate certificate and the leaf certificate).

kubectl create secret tls goecho-tls --key ./certs/leaf.key --cert ./certs/chain.crt

Next, patch the echo deployment to use the secret and serve HTTPS using it.

kubectl patch deployment echo -p '{
  "spec": {
    "template": {
      "spec": {
        "containers": [
          {
            "name": "echo",
            "ports": [
              {
                "containerPort": 443
              }
            ],
            "env": [
              {
                "name": "HTTPS_PORT",
                "value": "443"
              },
              {
                "name": "TLS_CERT_FILE",
                "value": "/etc/tls/tls.crt"
              },
              {
                "name": "TLS_KEY_FILE",
                "value": "/etc/tls/tls.key"
              }
            ],
            "volumeMounts": [
              {
                "mountPath": "/etc/tls",
                "name": "tls"
              }
            ]
          }
        ],
        "volumes": [
          {
            "name": "tls",
            "secret": {
              "secretName": "goecho-tls"
            }
          }
        ]
      }
    }
  }
}'

The results should look like this:

deployment.apps/echo patched

Also, patch the service to use HTTPS by adding the konghq.com/protocol: https annotation and the spec.ports entry.

kubectl patch service echo -p '{
  "metadata": {
    "annotations": {
      "konghq.com/protocol": "https"
    }
  },
  "spec": {
    "ports": [
      {
        "name": "https",
        "port": 443,
        "targetPort": 443
      }
    ]
  }
}'

The results should look like this:

service/echo patched

Expose the goecho service

Now that the kong/goecho service is serving HTTPS, we need to expose it.

Gateway API

Ingress

echo "
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: echo
  annotations:
    konghq.com/strip-path: 'true'
spec:
  parentRefs:
  - name: kong
  hostnames:
  - 'kong.example'
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /echo
    backendRefs:
    - name: echo
      kind: Service
      port: 443
" | kubectl apply -f -

echo "
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: echo
  annotations:
    konghq.com/strip-path: 'true'
spec:
  ingressClassName: kong
  rules:
  - host: kong.example
    http:
      paths:
      - path: /echo
        pathType: ImplementationSpecific
        backend:
          service:
            name: echo
            port:
              number: 443
" | kubectl apply -f -

The results should look like this:

      Gateway API
    
      Ingress
    
httproute.gateway.networking.k8s.io/echo created
ingress.networking.k8s.io/echo created

Verify connectivity by issuing an HTTP request to proxy. The service serves HTTPS but Kong Gateway initiates the connection and proxies it as HTTP in this case, thus the request should be made over HTTP. The Host header is required to match the hostname of the service.

curl -H Host:kong.example $PROXY_IP/echo

You should see a response similar to this:

Welcome, you are connected to node orbstack.
Running on Pod echo-bd94b7dcc-qxs2b.
In namespace default.
With IP address 192.168.194.9.
Through HTTPS connection.

That means the service is up and running and Kong Gateway connects to it successfully over HTTPS, without verification.

Configure Kong Gateway to verify the upstream TLS certificate

Enable TLS verification

Gateway API

Ingress

To configure Kong Gateway to verify the certificate of the upstream service, we need to create a BackendTLSPolicy resource:

Secret

ConfigMap

echo 'apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: goecho-tls-policy
spec:
  options:
    tls-verify-depth: "1"
  targetRefs:
  - group: core
    kind: Service
    name: echo
  validation:
    caCertificateRefs:
    - group: core
      kind: Secret
      name: root-ca
    hostname: kong.example' | kubectl apply -f -

echo 'apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: goecho-tls-policy
spec:
  options:
    tls-verify-depth: "1"
  targetRefs:
  - group: core
    kind: Service
    name: echo
  validation:
    caCertificateRefs:
    - group: core
      kind: ConfigMap
      name: root-ca
    hostname: kong.example' | kubectl apply -f -

The results should look like this.

backendtlspolicy.gateway.networking.k8s.io/goecho-tls-policy created

To configure Kong Gateway to verify the certificate of the upstream service, we need to annotate the service accordingly.

kubectl annotate service echo konghq.com/tls-verify=true

The results should look like this.

service/echo annotated

Now, when you issue the same request as before, you should see an error similar to this.

curl -H Host:kong.example $PROXY_IP/echo

{
  "message":"An invalid response was received from the upstream server",
  "request_id":"e2b3182856c96c23d61e880d0a28012f"
}

If you inspect Kong Gateway’s container logs, you should see an error indicating an issue with TLS handshake.

kubectl logs -n kong deploy/kong-gateway | grep "GET /echo"

2024/11/28 18:04:05 [error] 1280#0: *20004 upstream SSL certificate verify error: (20:unable to get local issuer certificate) while SSL handshaking to upstream, client: 192.168.194.1, server: kong, request: "GET /echo HTTP/1.1", upstream: "https://192.168.194.9:443/", host: "kong.example", request_id: "e2b3182856c96c23d61e880d0a28012f"
192.168.194.1 - - [28/Nov/2024:18:04:05 +0000] "GET /echo HTTP/1.1" 502 126 "-" "curl/8.7.1" kong_request_id: "e2b3182856c96c23d61e880d0a28012f"

Kong Gateway is now verifying the certificate of the upstream service and rejecting the connection because the certificate is not trusted.

Add the root CA certificate to Kong Gateway

That can be fixed by adding the root CA certificate to the Kong Gateway’s CA certificates and associating it with the service.

Gateway API

Ingress

Secret

ConfigMap

First, create a Secret with the root CA certificate.

kubectl create secret generic root-ca \
  --from-file=ca.crt=./certs/root.crt \
  --from-literal=id=bf6e0f14-78cd-45ad-9325-87ec7ef7b892 # An arbitrary ID for the certificate
kubectl label secret root-ca konghq.com/ca-cert=true # This label is required for the CA certificate to be recognized by Kong
kubectl annotate secret root-ca kubernetes.io/ingress.class=kong

The results should look like this.

secret/root-ca created
secret/root-ca labeled
secret/root-ca annotated

First, create a ConfigMap with the root CA certificate.

kubectl create configmap root-ca \
  --from-file=ca.crt=./certs/root.crt \
  --from-literal=id=bf6e0f14-78cd-45ad-9325-87ec7ef7b891 # An arbitrary ID for the certificate
kubectl label configmap root-ca konghq.com/ca-cert=true # This label is required for the CA certificate to be recognized by Kong
kubectl annotate configmap root-ca kubernetes.io/ingress.class=kong

The results should look like this.

configmap/root-ca created
configmap/root-ca labeled
configmap/root-ca annotated

The CA is already associated with the Service through BackendTLSPolicy’s spec.validation.caCertificateRefs.

Secret

ConfigMap

First, create a Secret with the root CA certificate.

kubectl create secret generic root-ca \
  --from-file=ca.crt=./certs/root.crt \
  --from-literal=id=bf6e0f14-78cd-45ad-9325-87ec7ef7b892 # An arbitrary ID for the certificate
kubectl label secret root-ca konghq.com/ca-cert=true # This label is required for the CA certificate to be recognized by Kong
kubectl annotate secret root-ca kubernetes.io/ingress.class=kong

The results should look like this.

secret/root-ca created
secret/root-ca labeled
secret/root-ca annotated

Now, associate the root CA certificate with the Service passing its name to konghq.com/ca-certificates-secrets annotation.

The konghq.com/ca-certificates-secrets annotation is a comma-separated list of Secrets holding CA certificates. You can add multiple Secrets to the list.

kubectl annotate service echo konghq.com/ca-certificates-secrets='root-ca'

The result should look like this.

service/echo annotated

First, create a ConfigMap with the root CA certificate.

kubectl create configmap root-ca \
  --from-file=ca.crt=./certs/root.crt \
  --from-literal=id=bf6e0f14-78cd-45ad-9325-87ec7ef7b891 # An arbitrary ID for the certificate
kubectl label configmap root-ca konghq.com/ca-cert=true # This label is required for the CA certificate to be recognized by Kong
kubectl annotate configmap root-ca kubernetes.io/ingress.class=kong

The results should look like this.

configmap/root-ca created
configmap/root-ca labeled
configmap/root-ca annotated

Now, associate the root CA certificate with the Service passing its name to konghq.com/ca-certificates-configmaps annotation.

The konghq.com/ca-certificates-configmaps annotation is a comma-separated list of ConfigMaps holding CA certificates. You can add multiple ConfigMaps to the list.

kubectl annotate service echo konghq.com/ca-certificates-configmaps='root-ca'

The result should look like this.

service/echo annotated

Now, when you issue the same request as before, you should see a successful response.

curl -H Host:kong.example $PROXY_IP/echo

Welcome, you are connected to node orbstack.
Running on Pod echo-bd94b7dcc-gdnhl.
In namespace default.
With IP address 192.168.194.18.
Through HTTPS connection.

Kong Gateway by default keeps upstream connections alive. By default this setting - upstream_keepalive_idle_timeout - is 60 (seconds). Because of that you might need to wait for 60 seconds to see the results. If you want to speed up the process, you can restart the Kong Gateway pod.

Kong Gateway is now verifying the certificate of the upstream service and accepting the connection because the certificate is trusted.

Configure verification depth

By default, Kong Gateway verifies the certificate chain up to the root CA certificate with no depth limit. You can configure the verification depth by annotating the service with the konghq.com/tls-verify-depth annotation.

Gateway API

Ingress

For example, to limit the verification depth to 1 (i.e., only verify one intermediate certificate), you can set the tls-verify-depth option in the BackendTLSPolicy resource like this:

kubectl patch backendtlspolicies.gateway.networking.k8s.io goecho-tls-policy --type merge -p='{
  "spec": {
    "options" : {
      "tls-verify-depth": "1"
    }
  }
}'

The results should look like this.

backendtlspolicy.gateway.networking.k8s.io/goecho-tls-policy patched

For example, to limit the verification depth to 1 (i.e., only verify one intermediate certificate), you can annotate the service like this:

kubectl annotate service echo konghq.com/tls-verify-depth=1

The results should look like this.

service/echo annotated

Now, when you issue the same request as before, you should still see a successful response.

curl -H Host:kong.example $PROXY_IP/echo

Welcome, you are connected to node orbstack.
Running on Pod echo-bd94b7dcc-gdnhl.
In namespace default.
With IP address
Through HTTPS connection.

You can also set the verification depth to 0 to not allow any intermediate certificates.

Gateway API

Ingress

kubectl patch backendtlspolicies.gateway.networking.k8s.io goecho-tls-policy --type merge -p='{
  "spec": {
    "options" : {
      "tls-verify-depth": "0"
    }
  }
}'

The results should look like this.

backendtlspolicy.gateway.networking.k8s.io/goecho-tls-policy patched

kubectl annotate --overwrite service echo konghq.com/tls-verify-depth=0

The results should look like this.

service/echo annotated

Now, when you issue the same request as before, you should see an error similar to this.

curl -H Host:kong.example $PROXY_IP/echo

{
  "message":"An invalid response was received from the upstream server",
  "request_id":"e2b3182856c96c23d61e880d0a28012f"
}

You can inspect Kong Gateway’s container logs to see the error.

kubectl logs -n kong deploy/kong-gateway | grep "GET /echo"

2024/11/29 11:41:46 [error] 1280#0: *45531 upstream SSL certificate verify error: (22:certificate chain too long) while SSL handshaking to upstream, client: 192.168.194.1, server: kong, request: "GET /echo HTTP/1.1", upstream: "https://192.168.194.19:443/", host: "kong.example", request_id: "678281372fb8907ed06d517cf515de78"
192.168.194.1 - - [29/Nov/2024:11:41:46 +0000] "GET /echo HTTP/1.1" 502 126 "-" "curl/8.7.1" kong_request_id: "678281372fb8907ed06d517cf515de78"

Kong Gateway is now rejecting the connection because the certificate chain is too long. Changing the verification depth to 1 should allow the connection to succeed again.

Gateway API

Ingress

kubectl patch backendtlspolicies.gateway.networking.k8s.io goecho-tls-policy --type merge -p='{
  "spec": {
    "options" : {
      "tls-verify-depth": "1"
    }
  }
}'

The results should look like this.

backendtlspolicy.gateway.networking.k8s.io/goecho-tls-policy patched

For example, to limit the verification depth to 1 (i.e., only verify one intermediate certificate), you can annotate the service like this:

kubectl annotate --overwrite service echo konghq.com/tls-verify-depth=1

The results should look like this.

service/echo annotated

Now, when you issue the same request as before, you should see a successful response.

curl -H Host:kong.example $PROXY_IP/echo

Welcome, you are connected to node orbstack.
Running on Pod echo-bd94b7dcc-9lvmf.
In namespace default.
With IP address 192.168.194.19.
Through HTTPS connection.