Securing Cilium's Gateway Api with cert-manager

Estimated reading time: 9 minutes

Gerard Samuel Gerard Samuel's profile photo

In my Hashicorp Nomad cluster, I am using Traefik to proxy external connections to the running containers, and Traefik also terminates TLS connections. While it is perfectly okay to duplicate this role in Kubernetes, I decided to go another route and leverage Gateway API as the reverse proxy. To build upon my existing work with Gateway API, let me set up an HTTP/HTTPS proxy with redirection using Gateway API and secure it with cert-manager and a few friends.

cert-manager, as described on their website, is a tool that helps manage and automate the issuance and renewal of TLS certificates in Kubernetes environments. cert-manager can obtain certificates from a variety of certificate authorities, including: Let’s Encrypt, HashiCorp Vault, Venafi and private PKI.

For my use case, here are the requirements that are needed to support cert-manager:

  1. A PKI Certificate Authority. I already have internal PKI services with step-ca, that also has ACME protocol support.
  2. A mechanism to complete ACME challenges. cert-manager supports both HTTP-01 and DNS-01 challenge types. I used DNS-01 with my internal Technitium DNS servers with support for RFC-2136 dynamic zone updates.
  3. A Kubernetes cluster with Cilium CNI installed, with its BGP and Gateway API enabled.

Technitium DNS

I need to configure DNS so that cert-manager can modify DNS records dynamically. To do this, I will create a transaction signature (TSIG) secret and grant the holder of this secret the right to modify TXT records on the DNS server.

Log into your DNS server and navigate to Settings -> TSIG. Click Add

Technitium DNS settings menu
Technitium DNS settings menu

Give the TSIG key the name of cert-manager-tsig-key, choose HMAC-SHA512 for the algorithm, and click on Save Settings to auto-generate a strong key.

Creating a new TSIG secret
Creating a new TSIG secret

Note

The shared secret in the DNS portal will appear after saving the configuration.

Go to Zones -> <YOUR-DNS-ZONE> -> Options -> Zone Options

Technitium DNS Zone Options menu
Technitium DNS Zone Options menu

Go to the new window’s Dynamic Updates (RFC 2136) tab. Scroll to the bottom in the Security Policy section. Select the newly created TSIG key name, enter a domain as a wildcard (i.e., *.tld), and choose the TXT record type. Click Save.

RFC-2136 zone options
RFC-2136 zone options

This configuration will allow any holder of the shared key to modify TXT records in the zone. An IP allowlist for Dynamic Updates (RFC 2136) can be defined on this same page.

Step-ca

Since I have a private CA, I need to provide a copy of the root certificate to Kubernetes to form a trust.

SSH into your step-ca server, and export the path for step-ca: export STEPPATH="/etc/step-ca".

Get a base64 encoded string of the root certificate using the step cli: sudo -E step ca root | sudo step base64.

Cert-Manager

Install cert-manager with its Gateway API feature set enabled.

helm repo add jetstack https://charts.jetstack.io --force-update
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.17.0 \
  --set crds.enabled=true \
  --set config.apiVersion="controller.config.cert-manager.io/v1alpha1" \
  --set config.kind="ControllerConfiguration" \
  --set config.enableGatewayAPI=true

Verify that cert-manager is operating correctly.

❯ kubectl get pods -n cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-6fc5d96984-f6hgf              1/1     Running   0          81s
cert-manager-cainjector-7c8f7984fb-glhtt   1/1     Running   0          81s
cert-manager-webhook-7594bcdb99-6thwh      1/1     Running   0          81s

Create a Kubernetes secret using the TSIG shared key from Technitium DNS under .stringData.

# dns-secret.yaml
tee ./infra/apps/cert-manager/dns-secret.yaml << EOF > /dev/null
---
apiVersion: v1
kind: Secret
metadata:
  name: technitium-tsig-secret
type: Opaque
stringData:
  tsig-key: <ENTER-YOUR-TSIG-KEY-HERE>
EOF

Create a ClusterIssuer manifest using the base64 encoded string of step-ca’s root certificate.

# cluster-issuer.yaml
tee ./infra/apps/cert-manager/cluster-issuer.yaml << EOF > /dev/null
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: acme-cluster-issuer
spec:
  acme:
    email: [email protected]
    server: https://ca.lab.howto.engineer/acme/acme/directory
    caBundle: <ENTER-YOUR-CA-ROOT-BASE64-ENCODED-HERE>
    privateKeySecretRef:
      name: acme-issuer-account-key
    solvers:
    - dns01:
        rfc2136:
          nameserver: 192.168.108.10:53
          tsigKeyName: cert-manager
          tsigAlgorithm: HMACSHA512
          tsigSecretSecretRef:
            name: technitium-tsig-secret
            key: tsig-key
      selector:
        dnsZones:
          - lab.howto.engineer
          - '*.lab.howto.engineer'
EOF

Sections of this file to pay attention to:

  • .spec.acme.server should point to the ACME endpoint
  • .spec.acme.caBundle is where to paste in the base64 encoded string of your root certificate
  • .spec.acme.solvers[*].rfc2136 should match the configuration of the TSIG shared key created on the DNS server
  • .spec.acme.solvers.selector.dnsZones should match your FQDN that your PKI trusts

Deploy the DNS secret and ClusterIssuer.

kubectl apply -n cert-manager \
  -f ./infra/apps/cert-manager/dns-secret.yaml \
  -f ./infra/apps/cert-manager/cluster-issuer.yaml

Gateway API

Create a namespace and a Gateway.

# infra/cilium/gateway.yaml
tee ./infra/cilium/gateway.yaml << EOF > /dev/null
---
apiVersion: v1
kind: Namespace
metadata:
  name: infra-gateway
  labels:
    name: infra
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-gateway
  namespace: infra-gateway
  annotations:
    cert-manager.io/cluster-issuer: acme-cluster-issuer
spec:
  gatewayClassName: cilium
  listeners:
  - name: http
    port: 80
    protocol: HTTP
    hostname: '*.lab.howto.engineer'
    allowedRoutes:
      namespaces:
        from: All    
  - name: https
    port: 443
    protocol: HTTPS
    hostname: '*.lab.howto.engineer'
    allowedRoutes:
      namespaces:
        from: All
    tls:
      certificateRefs:
        - kind: Secret
          name: lab-howto-engineer-tls
EOF

Areas of interest in this file are:

  • .metadata.annotations.cert-manager.io/cluster-issuer should be set to the ClusterIssuer created earlier.
  • .spec.listeners[*].hostname should set a wildcard value of your FQDN, so that it matches valid names such as example.tld or sub.example.tld.
  • .spec.listeners.tls.certificateRefs[*].name is an arbitrary name for the to-be generated certificate.

Deploy the Gateway.

kubectl apply -f ./infra/cilium/gateway.yaml

Validation

Wait a minute or two for the cert-manager to complete the DNS challenge and get an ACME certificate from the CA. Check the certificate’s state by running kubectl get certificate/lab-howto-engineer-tls -n infra-gateway.

❯ kubectl get certificate/lab-howto-engineer-tls -n infra-gateway
NAME                     READY   SECRET                   AGE
lab-howto-engineer-tls   False   lab-howto-engineer-tls   24s

In Technitium DNS, go to Logs -> Query Logs. Narrow down the timeframe, and for the domain, enter _acme-challenge.lab.howto.engineer. Click Query

Technitium DNS Logs
Technitium DNS Logs

Here, I can see that a DNS challenge occurred without any errors.

Workload testing

Let’s deploy a workload to test end-to-end connectivity.

# whoami.yaml
tee whoami.yaml << EOF > /dev/null
---
apiVersion: v1
kind: Namespace
metadata:
  name: whoami
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: http-redirect
  namespace: whoami
spec:
  parentRefs:
  - name: my-gateway
    namespace: infra-gateway
    sectionName: http
  rules:
  - filters:
    - type: RequestRedirect
      requestRedirect:
        scheme: https
        statusCode: 301
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httproute-whoami
  namespace: whoami
spec:
  parentRefs:
  - name: my-gateway
    namespace: infra-gateway
    sectionName: https
  hostnames:
  - "whoami.lab.howto.engineer"
  rules:
  - backendRefs:
    - name: whoami
      port: 80
---
apiVersion: v1
kind: Service
metadata:
  name: whoami
  namespace: whoami
  labels:
    app: whoami
    service: whoami
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: whoami
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami-http
  namespace: whoami
spec:
  replicas: 3
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: traefik/whoami
          ports:
            - name: web
              containerPort: 80
EOF

Let me zoom into the two HTTPRoute objects in this file.

Both HTTPRoute objects refer to the created gateway in a specific namespace under .spec.parentRefs[*].

The first HTTPRoute redirects any HTTP connections to HTTPS, defined under .spec.rules[*].filters.

The second HTTPRoute will route connections to the HTTP hostnames listed under .spec.hostnames and send the connection to a service under .spec.rules[*].backendRefs.

Deploy the manifests.

kubectl apply -f whoami.yaml

Create a DNS entry for whoami.lab.howto.engineer from the output of kubectl get gateway/my-gateway -n infra-gateway -o jsonpath='{.status.addresses[0].value}'.

An HTTPS test:

❯ curl -s https://whoami.lab.howto.engineer/api | jq '.'
{
  "hostname": "whoami-http-68965c9df8-qhwsz",
  "ip": [
    "127.0.0.1",
    "::1",
    "10.244.1.138",
    "fe80::3061:e7ff:fea3:4d8f"
  ],
  "headers": {
    "Accept": [
      "*/*"
    ],
    "User-Agent": [
      "curl/8.7.1"
    ],
    "X-Envoy-Internal": [
      "true"
    ],
    "X-Forwarded-For": [
      "192.168.20.93"
    ],
    "X-Forwarded-Proto": [
      "https"
    ],
    "X-Request-Id": [
      "f9a0fdb8-6732-4efd-b23c-672de4f84972"
    ]
  },
  "url": "/api",
  "host": "whoami.lab.howto.engineer",
  "method": "GET",
  "remoteAddr": "10.244.5.53:37195"
}

An HTTP test (with redirection to HTTPS):

❯ curl -sL http://whoami.lab.howto.engineer/api | jq '.'
{
  "hostname": "whoami-http-68965c9df8-bt242",
  "ip": [
    "127.0.0.1",
    "::1",
    "10.244.5.90",
    "fe80::24ca:48ff:fee2:21f7"
  ],
  "headers": {
    "Accept": [
      "*/*"
    ],
    "User-Agent": [
      "curl/8.7.1"
    ],
    "X-Envoy-Internal": [
      "true"
    ],
    "X-Forwarded-For": [
      "192.168.20.93"
    ],
    "X-Forwarded-Proto": [
      "https"
    ],
    "X-Request-Id": [
      "5fec95dd-04aa-4fde-957a-b70ed9e133dc"
    ]
  },
  "url": "/api",
  "host": "whoami.lab.howto.engineer",
  "method": "GET",
  "remoteAddr": "10.244.5.53:39249"
}

Now let’s review HTTP headers while redirecting from HTTP to HTTPS:

❯ curl -sLv http://whoami.lab.howto.engineer/api -o /dev/null
* Host whoami.lab.howto.engineer:80 was resolved.
* IPv6: (none)
* IPv4: 192.168.254.10
*   Trying 192.168.254.10:80...
* Connected to whoami.lab.howto.engineer (192.168.254.10) port 80
> GET /api HTTP/1.1
> Host: whoami.lab.howto.engineer
> User-Agent: curl/8.7.1
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 301 Moved Permanently
< location: https://whoami.lab.howto.engineer:443/api
< date: Fri, 07 Feb 2025 02:17:40 GMT
< server: envoy
< content-length: 0
< 
* Ignoring the response-body
* Connection #0 to host whoami.lab.howto.engineer left intact
* Clear auth, redirects to port from 80 to 443
* Issue another request to this URL: 'https://whoami.lab.howto.engineer:443/api'
* Host whoami.lab.howto.engineer:443 was resolved.
* IPv6: (none)
* IPv4: 192.168.254.10
*   Trying 192.168.254.10:443...
* Connected to whoami.lab.howto.engineer (192.168.254.10) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
} [330 bytes data]
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* (304) (IN), TLS handshake, Unknown (8):
{ [15 bytes data]
* (304) (IN), TLS handshake, Certificate (11):
{ [1196 bytes data]
* (304) (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* (304) (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* (304) (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
*  subject: [NONE]
*  start date: Feb  6 19:44:32 2025 GMT
*  expire date: Feb  7 19:45:32 2025 GMT
*  subjectAltName: host "whoami.lab.howto.engineer" matched cert's "*.lab.howto.engineer"
*  issuer: CN=HTE Intermediate CA
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://whoami.lab.howto.engineer:443/api
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: whoami.lab.howto.engineer]
* [HTTP/2] [1] [:path: /api]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET /api HTTP/2
> Host: whoami.lab.howto.engineer
> User-Agent: curl/8.7.1
> Accept: */*
> 
* Request completely sent off
< HTTP/2 200 
< content-type: application/json
< date: Fri, 07 Feb 2025 02:17:40 GMT
< content-length: 414
< x-envoy-upstream-service-time: 1
< server: envoy
< 
{ [414 bytes data]
* Connection #1 to host whoami.lab.howto.engineer left intact

Conclusion

In this post, I configured a DNS server with RFC-2136 protocol support for DNS-01 ACME challenges, cert-manager, and Cilium’s Gateway API to create a reverse proxy secured with an auto-renewing certificate. Following the Gateway API specification, one can do more than just basic routing and redirection. Here are a couple of other things that you can do with Gateway API:

One thing that I want to improve on is using sealed secrets for the DNS TSIG shared key.

While this may not have all the bells and whistles of a full-fledged reverse proxy such as Traefik, I can use the process in this article to cover most of the use cases Traefik does for my environment.

Thanks!

See Also