[{"content":"","date":"2025-02","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","date":"2025-02","externalUrl":null,"permalink":"/tags/cert-manager/","section":"Tags","summary":"","title":"Cert-Manager","type":"tags"},{"content":"","date":"2025-02","externalUrl":null,"permalink":"/tags/cilium/","section":"Tags","summary":"","title":"Cilium","type":"tags"},{"content":"","date":"2025-02","externalUrl":null,"permalink":"/tags/gateway-api/","section":"Tags","summary":"","title":"Gateway-Api","type":"tags"},{"content":"","date":"2025-02","externalUrl":null,"permalink":"/","section":"Gerard Samuel","summary":"","title":"Gerard Samuel","type":"page"},{"content":"","date":"2025-02","externalUrl":null,"permalink":"/categories/homelab/","section":"Categories","summary":"","title":"Homelab","type":"categories"},{"content":"","date":"2025-02","externalUrl":null,"permalink":"/posts/","section":"Posts","summary":"","title":"Posts","type":"posts"},{"content":"In my Hashicorp Nomad cluster, I am using Traefik to proxy external connections to the running containers, and Traefik also terminates TLS connections. While it is perfectly okay to duplicate this role in Kubernetes, I decided to go another route and leverage Gateway API as the reverse proxy. To build upon my existing work with Gateway API, let me set up an HTTP/HTTPS proxy with redirection using Gateway API and secure it with cert-manager and a few friends.\ncert-manager, as described on their website, is a tool that helps manage and automate the issuance and renewal of TLS certificates in Kubernetes environments. cert-manager can obtain certificates from a variety of certificate authorities, including: Let\u0026rsquo;s Encrypt, HashiCorp Vault, Venafi and private PKI.\nFor my use case, here are the requirements that are needed to support cert-manager:\nA PKI Certificate Authority. I already have internal PKI services with step-ca, that also has ACME protocol support. A mechanism to complete ACME challenges. cert-manager supports both HTTP-01 and DNS-01 challenge types. I used DNS-01 with my internal Technitium DNS servers with support for RFC-2136 dynamic zone updates. A Kubernetes cluster with Cilium CNI installed, with its BGP and Gateway API enabled. Technitium DNS # I need to configure DNS so that cert-manager can modify DNS records dynamically. To do this, I will create a transaction signature (TSIG) secret and grant the holder of this secret the right to modify TXT records on the DNS server.\nLog into your DNS server and navigate to Settings -\u0026gt; TSIG. Click Add Technitium DNS settings menu Give the TSIG key the name of cert-manager-tsig-key, choose HMAC-SHA512 for the algorithm, and click on Save Settings to auto-generate a strong key. Creating a new TSIG secret Note The shared secret in the DNS portal will appear after saving the configuration.\nGo to Zones -\u0026gt; \u0026lt;YOUR-DNS-ZONE\u0026gt; -\u0026gt; Options -\u0026gt; Zone Options Technitium DNS Zone Options menu Go to the new window\u0026rsquo;s Dynamic Updates (RFC 2136) tab. Scroll to the bottom in the Security Policy section. Select the newly created TSIG key name, enter a domain as a wildcard (i.e., *.tld), and choose the TXT record type. Click Save. RFC-2136 zone options This configuration will allow any holder of the shared key to modify TXT records in the zone. An IP allowlist for Dynamic Updates (RFC 2136) can be defined on this same page.\nStep-ca # Since I have a private CA, I need to provide a copy of the root certificate to Kubernetes to form a trust.\nSSH into your step-ca server, and export the path for step-ca: export STEPPATH=\u0026quot;/etc/step-ca\u0026quot;.\nGet a base64 encoded string of the root certificate using the step cli: sudo -E step ca root | sudo step base64.\nCert-Manager # Install cert-manager with its Gateway API feature set enabled.\nhelm repo add jetstack https://charts.jetstack.io --force-update helm install \\ cert-manager jetstack/cert-manager \\ --namespace cert-manager \\ --create-namespace \\ --version v1.17.0 \\ --set crds.enabled=true \\ --set config.apiVersion=\u0026#34;controller.config.cert-manager.io/v1alpha1\u0026#34; \\ --set config.kind=\u0026#34;ControllerConfiguration\u0026#34; \\ --set config.enableGatewayAPI=true Verify that cert-manager is operating correctly.\n❯ kubectl get pods -n cert-manager NAME READY STATUS RESTARTS AGE cert-manager-6fc5d96984-f6hgf 1/1 Running 0 81s cert-manager-cainjector-7c8f7984fb-glhtt 1/1 Running 0 81s cert-manager-webhook-7594bcdb99-6thwh 1/1 Running 0 81s Create a Kubernetes secret using the TSIG shared key from Technitium DNS under .stringData.\n# dns-secret.yaml tee ./infra/apps/cert-manager/dns-secret.yaml \u0026lt;\u0026lt; EOF \u0026gt; /dev/null --- apiVersion: v1 kind: Secret metadata: name: technitium-tsig-secret type: Opaque stringData: tsig-key: \u0026lt;ENTER-YOUR-TSIG-KEY-HERE\u0026gt; EOF Create a ClusterIssuer manifest using the base64 encoded string of step-ca\u0026rsquo;s root certificate.\n# cluster-issuer.yaml tee ./infra/apps/cert-manager/cluster-issuer.yaml \u0026lt;\u0026lt; EOF \u0026gt; /dev/null --- apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: acme-cluster-issuer spec: acme: email: your-name@example.com server: https://ca.lab.howto.engineer/acme/acme/directory caBundle: \u0026lt;ENTER-YOUR-CA-ROOT-BASE64-ENCODED-HERE\u0026gt; privateKeySecretRef: name: acme-issuer-account-key solvers: - dns01: rfc2136: nameserver: 192.168.108.10:53 tsigKeyName: cert-manager tsigAlgorithm: HMACSHA512 tsigSecretSecretRef: name: technitium-tsig-secret key: tsig-key selector: dnsZones: - lab.howto.engineer - \u0026#39;*.lab.howto.engineer\u0026#39; EOF Sections of this file to pay attention to:\n.spec.acme.server should point to the ACME endpoint .spec.acme.caBundle is where to paste in the base64 encoded string of your root certificate .spec.acme.solvers[*].rfc2136 should match the configuration of the TSIG shared key created on the DNS server .spec.acme.solvers.selector.dnsZones should match your FQDN that your PKI trusts Deploy the DNS secret and ClusterIssuer.\nkubectl apply -n cert-manager \\ -f ./infra/apps/cert-manager/dns-secret.yaml \\ -f ./infra/apps/cert-manager/cluster-issuer.yaml Gateway API # Create a namespace and a Gateway.\n# infra/cilium/gateway.yaml tee ./infra/cilium/gateway.yaml \u0026lt;\u0026lt; EOF \u0026gt; /dev/null --- apiVersion: v1 kind: Namespace metadata: name: infra-gateway labels: name: infra --- apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: my-gateway namespace: infra-gateway annotations: cert-manager.io/cluster-issuer: acme-cluster-issuer spec: gatewayClassName: cilium listeners: - name: http port: 80 protocol: HTTP hostname: \u0026#39;*.lab.howto.engineer\u0026#39; allowedRoutes: namespaces: from: All - name: https port: 443 protocol: HTTPS hostname: \u0026#39;*.lab.howto.engineer\u0026#39; allowedRoutes: namespaces: from: All tls: certificateRefs: - kind: Secret name: lab-howto-engineer-tls EOF Areas of interest in this file are:\n.metadata.annotations.cert-manager.io/cluster-issuer should be set to the ClusterIssuer created earlier. .spec.listeners[*].hostname should set a wildcard value of your FQDN, so that it matches valid names such as example.tld or sub.example.tld. .spec.listeners.tls.certificateRefs[*].name is an arbitrary name for the to-be generated certificate. Deploy the Gateway.\nkubectl apply -f ./infra/cilium/gateway.yaml Validation # Wait a minute or two for the cert-manager to complete the DNS challenge and get an ACME certificate from the CA. Check the certificate\u0026rsquo;s state by running kubectl get certificate/lab-howto-engineer-tls -n infra-gateway.\n❯ kubectl get certificate/lab-howto-engineer-tls -n infra-gateway NAME READY SECRET AGE lab-howto-engineer-tls False lab-howto-engineer-tls 24s In Technitium DNS, go to Logs -\u0026gt; Query Logs. Narrow down the timeframe, and for the domain, enter _acme-challenge.lab.howto.engineer. Click Query Technitium DNS Logs Here, I can see that a DNS challenge occurred without any errors.\nWorkload testing # Let\u0026rsquo;s deploy a workload to test end-to-end connectivity.\n# whoami.yaml tee whoami.yaml \u0026lt;\u0026lt; EOF \u0026gt; /dev/null --- apiVersion: v1 kind: Namespace metadata: name: whoami --- apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: http-redirect namespace: whoami spec: parentRefs: - name: my-gateway namespace: infra-gateway sectionName: http rules: - filters: - type: RequestRedirect requestRedirect: scheme: https statusCode: 301 --- apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: httproute-whoami namespace: whoami spec: parentRefs: - name: my-gateway namespace: infra-gateway sectionName: https hostnames: - \u0026#34;whoami.lab.howto.engineer\u0026#34; rules: - backendRefs: - name: whoami port: 80 --- apiVersion: v1 kind: Service metadata: name: whoami namespace: whoami labels: app: whoami service: whoami spec: ports: - port: 80 name: http selector: app: whoami --- apiVersion: apps/v1 kind: Deployment metadata: name: whoami-http namespace: whoami spec: replicas: 3 selector: matchLabels: app: whoami template: metadata: labels: app: whoami spec: containers: - name: whoami image: traefik/whoami ports: - name: web containerPort: 80 EOF Let me zoom into the two HTTPRoute objects in this file.\nBoth HTTPRoute objects refer to the created gateway in a specific namespace under .spec.parentRefs[*].\nThe first HTTPRoute redirects any HTTP connections to HTTPS, defined under .spec.rules[*].filters.\nThe second HTTPRoute will route connections to the HTTP hostnames listed under .spec.hostnames and send the connection to a service under .spec.rules[*].backendRefs.\nDeploy the manifests.\nkubectl apply -f whoami.yaml Create a DNS entry for whoami.lab.howto.engineer from the output of kubectl get gateway/my-gateway -n infra-gateway -o jsonpath='{.status.addresses[0].value}'.\nAn HTTPS test:\n❯ curl -s https://whoami.lab.howto.engineer/api | jq \u0026#39;.\u0026#39; { \u0026#34;hostname\u0026#34;: \u0026#34;whoami-http-68965c9df8-qhwsz\u0026#34;, \u0026#34;ip\u0026#34;: [ \u0026#34;127.0.0.1\u0026#34;, \u0026#34;::1\u0026#34;, \u0026#34;10.244.1.138\u0026#34;, \u0026#34;fe80::3061:e7ff:fea3:4d8f\u0026#34; ], \u0026#34;headers\u0026#34;: { \u0026#34;Accept\u0026#34;: [ \u0026#34;*/*\u0026#34; ], \u0026#34;User-Agent\u0026#34;: [ \u0026#34;curl/8.7.1\u0026#34; ], \u0026#34;X-Envoy-Internal\u0026#34;: [ \u0026#34;true\u0026#34; ], \u0026#34;X-Forwarded-For\u0026#34;: [ \u0026#34;192.168.20.93\u0026#34; ], \u0026#34;X-Forwarded-Proto\u0026#34;: [ \u0026#34;https\u0026#34; ], \u0026#34;X-Request-Id\u0026#34;: [ \u0026#34;f9a0fdb8-6732-4efd-b23c-672de4f84972\u0026#34; ] }, \u0026#34;url\u0026#34;: \u0026#34;/api\u0026#34;, \u0026#34;host\u0026#34;: \u0026#34;whoami.lab.howto.engineer\u0026#34;, \u0026#34;method\u0026#34;: \u0026#34;GET\u0026#34;, \u0026#34;remoteAddr\u0026#34;: \u0026#34;10.244.5.53:37195\u0026#34; } An HTTP test (with redirection to HTTPS):\n❯ curl -sL http://whoami.lab.howto.engineer/api | jq \u0026#39;.\u0026#39; { \u0026#34;hostname\u0026#34;: \u0026#34;whoami-http-68965c9df8-bt242\u0026#34;, \u0026#34;ip\u0026#34;: [ \u0026#34;127.0.0.1\u0026#34;, \u0026#34;::1\u0026#34;, \u0026#34;10.244.5.90\u0026#34;, \u0026#34;fe80::24ca:48ff:fee2:21f7\u0026#34; ], \u0026#34;headers\u0026#34;: { \u0026#34;Accept\u0026#34;: [ \u0026#34;*/*\u0026#34; ], \u0026#34;User-Agent\u0026#34;: [ \u0026#34;curl/8.7.1\u0026#34; ], \u0026#34;X-Envoy-Internal\u0026#34;: [ \u0026#34;true\u0026#34; ], \u0026#34;X-Forwarded-For\u0026#34;: [ \u0026#34;192.168.20.93\u0026#34; ], \u0026#34;X-Forwarded-Proto\u0026#34;: [ \u0026#34;https\u0026#34; ], \u0026#34;X-Request-Id\u0026#34;: [ \u0026#34;5fec95dd-04aa-4fde-957a-b70ed9e133dc\u0026#34; ] }, \u0026#34;url\u0026#34;: \u0026#34;/api\u0026#34;, \u0026#34;host\u0026#34;: \u0026#34;whoami.lab.howto.engineer\u0026#34;, \u0026#34;method\u0026#34;: \u0026#34;GET\u0026#34;, \u0026#34;remoteAddr\u0026#34;: \u0026#34;10.244.5.53:39249\u0026#34; } Now let\u0026rsquo;s review HTTP headers while redirecting from HTTP to HTTPS:\n❯ curl -sLv http://whoami.lab.howto.engineer/api -o /dev/null * Host whoami.lab.howto.engineer:80 was resolved. * IPv6: (none) * IPv4: 192.168.254.10 * Trying 192.168.254.10:80... * Connected to whoami.lab.howto.engineer (192.168.254.10) port 80 \u0026gt; GET /api HTTP/1.1 \u0026gt; Host: whoami.lab.howto.engineer \u0026gt; User-Agent: curl/8.7.1 \u0026gt; Accept: */* \u0026gt; * Request completely sent off \u0026lt; HTTP/1.1 301 Moved Permanently \u0026lt; location: https://whoami.lab.howto.engineer:443/api \u0026lt; date: Fri, 07 Feb 2025 02:17:40 GMT \u0026lt; server: envoy \u0026lt; content-length: 0 \u0026lt; * Ignoring the response-body * Connection #0 to host whoami.lab.howto.engineer left intact * Clear auth, redirects to port from 80 to 443 * Issue another request to this URL: \u0026#39;https://whoami.lab.howto.engineer:443/api\u0026#39; * Host whoami.lab.howto.engineer:443 was resolved. * IPv6: (none) * IPv4: 192.168.254.10 * Trying 192.168.254.10:443... * Connected to whoami.lab.howto.engineer (192.168.254.10) port 443 * ALPN: curl offers h2,http/1.1 * (304) (OUT), TLS handshake, Client hello (1): } [330 bytes data] * CAfile: /etc/ssl/cert.pem * CApath: none * (304) (IN), TLS handshake, Server hello (2): { [122 bytes data] * (304) (IN), TLS handshake, Unknown (8): { [15 bytes data] * (304) (IN), TLS handshake, Certificate (11): { [1196 bytes data] * (304) (IN), TLS handshake, CERT verify (15): { [264 bytes data] * (304) (IN), TLS handshake, Finished (20): { [36 bytes data] * (304) (OUT), TLS handshake, Finished (20): } [36 bytes data] * SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF * ALPN: server accepted h2 * Server certificate: * subject: [NONE] * start date: Feb 6 19:44:32 2025 GMT * expire date: Feb 7 19:45:32 2025 GMT * subjectAltName: host \u0026#34;whoami.lab.howto.engineer\u0026#34; matched cert\u0026#39;s \u0026#34;*.lab.howto.engineer\u0026#34; * issuer: CN=HTE Intermediate CA * SSL certificate verify ok. * using HTTP/2 * [HTTP/2] [1] OPENED stream for https://whoami.lab.howto.engineer:443/api * [HTTP/2] [1] [:method: GET] * [HTTP/2] [1] [:scheme: https] * [HTTP/2] [1] [:authority: whoami.lab.howto.engineer] * [HTTP/2] [1] [:path: /api] * [HTTP/2] [1] [user-agent: curl/8.7.1] * [HTTP/2] [1] [accept: */*] \u0026gt; GET /api HTTP/2 \u0026gt; Host: whoami.lab.howto.engineer \u0026gt; User-Agent: curl/8.7.1 \u0026gt; Accept: */* \u0026gt; * Request completely sent off \u0026lt; HTTP/2 200 \u0026lt; content-type: application/json \u0026lt; date: Fri, 07 Feb 2025 02:17:40 GMT \u0026lt; content-length: 414 \u0026lt; x-envoy-upstream-service-time: 1 \u0026lt; server: envoy \u0026lt; { [414 bytes data] * Connection #1 to host whoami.lab.howto.engineer left intact Conclusion # In this post, I configured a DNS server with RFC-2136 protocol support for DNS-01 ACME challenges, cert-manager, and Cilium\u0026rsquo;s Gateway API to create a reverse proxy secured with an auto-renewing certificate. Following the Gateway API specification, one can do more than just basic routing and redirection. Here are a couple of other things that you can do with Gateway API:\nPath and hostname rewrites Request and Response header modifications Traffic splitting Terminated and passthrough TLS and more\u0026hellip; One thing that I want to improve on is using sealed secrets for the DNS TSIG shared key.\nWhile this may not have all the bells and whistles of a full-fledged reverse proxy such as Traefik, I can use the process in this article to cover most of the use cases Traefik does for my environment.\nThanks!\n","date":"2025-02","externalUrl":null,"permalink":"/posts/how-to-secure-cilium-gateway-api-with-cert-manager/","section":"Posts","summary":"In my Hashicorp Nomad cluster, I am using Traefik to proxy external connections to the running containers, and Traefik also terminates TLS connections. While it is perfectly okay to duplicate this role in Kubernetes, I decided to go another route and leverage Gateway API as the reverse proxy. To build upon my existing work with Gateway API, let me set up an HTTP/HTTPS proxy with redirection using Gateway API and secure it with cert-manager and a few friends.\n","title":"Securing Cilium's Gateway Api with cert-manager","type":"posts"},{"content":"","date":"2025-02","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"2025-01","externalUrl":null,"permalink":"/tags/bgp/","section":"Tags","summary":"","title":"Bgp","type":"tags"},{"content":"In my previous article on building a Kubernetes cluster with Talos Linux, I used a Kubernetes Service of type NodePort to expose a workload to my homelab network. However, exposing workloads using NodePorts is not efficient or standard practice. In this article, I will document how I configured Cilium\u0026rsquo;s Gateway API as a basic reverse proxy and BGP Control Plane to inject routing paths into the routing table of a UniFi router for the reverse proxy IP address.\nPrerequisites # Unifi UDM SE (or a supported device) running the UniFi OS v4.1+ 3 node control plane/worker Kubernetes version 1.32 cluster Cilium CNI version 1.16.5 installed All Kubernetes nodes and pods in a Ready/Running state kubectl version 1.32.0 installed locally Unifi # Create a FRR BGP configuration file for the BGP service in a UniFi router. Adjust router-id to match the IP address of your router. Adjust neighbor to match the IP addresses of the Kubernetes nodes. dev-clus-core is an arbitrary name I used to describe the Kubernetes cluster.\ntee bgp.conf \u0026lt;\u0026lt; EOF \u0026gt; /dev/null router bgp 65000 bgp bestpath as-path multipath-relax no bgp ebgp-requires-policy bgp router-id 192.168.2.1 neighbor dev-clus-core peer-group neighbor dev-clus-core remote-as external neighbor 192.168.110.31 peer-group dev-clus-core neighbor 192.168.110.32 peer-group dev-clus-core neighbor 192.168.110.33 peer-group dev-clus-core EOF Log into your UniFi Cloud Gateway and go to Settings -\u0026gt; Routing -\u0026gt; BGP. Enter a name for the configuration, select your router, and upload the bgp.conf file created earlier. Click Add That is it for BGP configuration on a UniFi router. Let me enable Cilium\u0026rsquo;s BGP Control Plane and Gateway API in Kubernetes.\nCilium BGP Control Plane # Install Cilium\u0026rsquo;s CLI for MacOS\nCILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt) CLI_ARCH=amd64 if [ \u0026#34;$(uname -m)\u0026#34; = \u0026#34;arm64\u0026#34; ]; then CLI_ARCH=arm64; fi curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-darwin-${CLI_ARCH}.tar.gz{,.sha256sum} shasum -a 256 -c cilium-darwin-${CLI_ARCH}.tar.gz.sha256sum sudo tar xzvfC cilium-darwin-${CLI_ARCH}.tar.gz /usr/local/bin rm cilium-darwin-${CLI_ARCH}.tar.gz{,.sha256sum} Install the required Gateway API CRDs (Custom Resource Definitions).\nkubectl apply \\ --filename https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml \\ --filename https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_gateways.yaml \\ --filename https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_httproutes.yaml \\ --filename https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_referencegrants.yaml \\ --filename https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/standard/gateway.networking.k8s.io_grpcroutes.yaml \\ --filename https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml Warning As of this writing, do not install version 1.2 of the Gateway API CRDs for Cilium 1.16.5 as it is incompatible/unsupported.\nUse version 1.1.0 of the Gateway API CRDs.\nEnable Cilium\u0026rsquo;s BGP Control Plane and Gateway API\ncilium upgrade --version 1.16.5 \\ --set bgpControlPlane.enabled=true \\ --set gatewayAPI.enabled=true \\ --set gatewayAPI.enableAlpn=true \\ --set gatewayAPI.enableAppProtocol=true After about a minute, check the state of Cilium by running cilium status --wait\n❯ cilium status --wait /¯¯\\ /¯¯\\__/¯¯\\ Cilium: OK \\__/¯¯\\__/ Operator: OK /¯¯\\__/¯¯\\ Envoy DaemonSet: OK \\__/¯¯\\__/ Hubble Relay: disabled \\__/ ClusterMesh: disabled DaemonSet cilium Desired: 3, Ready: 3/3, Available: 3/3 DaemonSet cilium-envoy Desired: 3, Ready: 3/3, Available: 3/3 Deployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1 Containers: cilium Running: 3 cilium-envoy Running: 3 cilium-operator Running: 1 Cluster Pods: 4/4 managed by Cilium Helm chart version: 1.16.5 Validate that the cilium gateway class was created by running kubectl get gatewayclass/cilium\n❯ kubectl get gatewayclass/cilium NAME CONTROLLER ACCEPTED AGE cilium io.cilium/gateway-controller True 3m47s Now I\u0026rsquo;ll create the BGP control plane manifests for CiliumBGPClusterConfig, CiliumBGPPeerConfig, CiliumBGPAdvertisement, and CiliumLoadBalancerIPPool\n# bgp-control-plane.yaml tee infra/cilium/bgp-control-plane.yaml \u0026lt;\u0026lt; EOF \u0026gt; /dev/null --- apiVersion: cilium.io/v2alpha1 kind: CiliumBGPClusterConfig metadata: name: cilium-bgp spec: nodeSelector: matchLabels: bgp: \u0026#34;65020\u0026#34; bgpInstances: - name: \u0026#34;65020\u0026#34; localASN: 65020 peers: - name: \u0026#34;udm-se-65000\u0026#34; peerASN: 65000 peerAddress: 192.168.2.1 peerConfigRef: name: \u0026#34;cilium-peer\u0026#34; --- apiVersion: cilium.io/v2alpha1 kind: CiliumBGPPeerConfig metadata: name: cilium-peer spec: gracefulRestart: enabled: true restartTimeSeconds: 15 families: - afi: ipv4 safi: unicast advertisements: matchLabels: advertise: \u0026#34;bgp\u0026#34; --- apiVersion: cilium.io/v2alpha1 kind: CiliumBGPAdvertisement metadata: name: bgp-advertisements labels: advertise: bgp spec: advertisements: - advertisementType: \u0026#34;Service\u0026#34; service: addresses: - LoadBalancerIP selector: matchExpressions: - {key: gateway.networking.k8s.io/gateway-name, operator: In, values: [\u0026#39;my-gateway\u0026#39;]} --- apiVersion: \u0026#34;cilium.io/v2alpha1\u0026#34; kind: CiliumLoadBalancerIPPool metadata: name: dev-core-lb-ip-pool spec: blocks: - start: \u0026#34;192.168.254.10\u0026#34; stop: \u0026#34;192.168.254.30\u0026#34; serviceSelector: matchExpressions: - {key: gateway.networking.k8s.io/gateway-name, operator: In, values: [\u0026#39;my-gateway\u0026#39;]} EOF Review this file as there a few items to pay attention to:\nFor CiliumBGPClusterConfig, .spec.nodeSelector requires labeling the nodes. For CiliumBGPClusterConfig, .spec.bgpInstances[*] needs to be configured for your router. For CiliumBGPAdvertisement, .spec.advertisements[*].selector needs to match the Gateway\u0026rsquo;s name (my-gateway), which I will create later. For CiliumLoadBalancerIPPool, .spec.blocks[*] can be configured with IP ranges or CIDRs ranges. For CiliumLoadBalancerIPPool, .spec.serviceSelector needs to match the Gateway\u0026rsquo;s name (my-gateway), which I will create later. Warning Do not pre-allocate network blocks in your router for CiliumLoadBalancerIPPool.spec.blocks[*]\nApply a label to the nodes so that it aligns with CiliumBGPClusterConfig.spec.nodeSelector\nkubectl label nodes --all bgp=65020 Apply ./infra/cilium/bgp-control-plane.yaml\nkubectl apply -f ./infra/cilium/bgp-control-plane.yaml Define a Gateway and HTTPRoute manifests\n# gateway.yaml tee ./infra/cilium/gateway.yaml \u0026lt;\u0026lt; EOF \u0026gt; /dev/null --- apiVersion: v1 kind: Namespace metadata: name: infra-gateway labels: name: infra --- apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: my-gateway namespace: infra-gateway spec: gatewayClassName: cilium listeners: - name: http protocol: HTTP port: 80 allowedRoutes: namespaces: from: All EOF Now apply the Gateway manifest to the cluster by running kubectl apply -f ./infra/cilium/gateway.yaml.\nReview the created gateway and associated service by running the following: kubectl get -n infra-gateway gateway,svc\n❯ kubectl get -n infra-gateway gateway,svc NAME CLASS ADDRESS PROGRAMMED AGE gateway.gateway.networking.k8s.io/my-gateway cilium 192.168.254.10 True 22s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/cilium-gateway-my-gateway LoadBalancer 10.107.142.196 192.168.254.10 80:30901/TCP 22s Notice the Address and External-IP columns respectively. An IP address was pulled from CiliumLoadBalancerIPPool.\nWith the pieces in place, let us take a look at BGP\u0026rsquo;s status. For Cilium, run the following two commands: cilium bgp peers \u0026amp;\u0026amp; cilium bgp routes\n❯ cilium bgp peers \u0026amp;\u0026amp; cilium bgp routes Node Local AS Peer AS Peer Address Session State Uptime Family Received Advertised dev-clus-core-cp01 65020 65000 192.168.2.1 established 4m5s ipv4/unicast 1 2 dev-clus-core-cp02 65020 65000 192.168.2.1 established 4m3s ipv4/unicast 1 2 dev-clus-core-cp03 65020 65000 192.168.2.1 established 4m2s ipv4/unicast 1 2 (Defaulting to `available ipv4 unicast` routes, please see help for more options) Node VRouter Prefix NextHop Age Attrs dev-clus-core-cp01 65020 192.168.254.10/32 0.0.0.0 57s [{Origin: i} {Nexthop: 0.0.0.0}] dev-clus-core-cp02 65020 192.168.254.10/32 0.0.0.0 57s [{Origin: i} {Nexthop: 0.0.0.0}] dev-clus-core-cp03 65020 192.168.254.10/32 0.0.0.0 57s [{Origin: i} {Nexthop: 0.0.0.0}] SSH into the router and review BGP\u0026rsquo;s status by running: vtysh -c 'show bgp summary' \u0026amp;\u0026amp; vtysh -c 'show ip bgp'\n# vtysh -c \u0026#39;show bgp summary\u0026#39; \u0026amp;\u0026amp; vtysh -c \u0026#39;show ip bgp\u0026#39; IPv4 Unicast Summary (VRF default): BGP router identifier 192.168.2.1, local AS number 65000 vrf-id 0 BGP table version 84 RIB entries 1, using 184 bytes of memory Peers 3, using 2169 KiB of memory Peer groups 1, using 64 bytes of memory Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc 192.168.110.31 4 65020 22595 22577 0 0 0 00:06:29 1 1 N/A 192.168.110.32 4 65020 22582 22537 0 0 0 00:06:27 1 1 N/A 192.168.110.33 4 65020 22579 22535 0 0 0 00:06:26 1 1 N/A Total number of neighbors 3 BGP table version is 84, local router ID is 192.168.2.1, vrf id 0 Default local pref 100, local AS 65000 Status codes: s suppressed, d damped, h history, * valid, \u0026gt; best, = multipath, i internal, r RIB-failure, S Stale, R Removed Nexthop codes: @NNN nexthop\u0026#39;s vrf id, \u0026lt; announce-nh-self Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found Network Next Hop Metric LocPrf Weight Path *\u0026gt; 192.168.254.10/32 192.168.110.31 0 65020 i *= 192.168.110.32 0 65020 i *= 192.168.110.33 0 65020 i Displayed 1 routes and 3 total paths From the router, also view the routing table by running netstat -ar\n# netstat -ar Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 10.255.253.0 0.0.0.0 255.255.255.0 U 0 0 0 br4040 \u0026lt;REDACT\u0026gt; 0.0.0.0 255.255.252.0 U 0 0 0 eth9 \u0026lt;REDACT\u0026gt; 0.0.0.0 255.255.255.192 U 0 0 0 eth8 192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 br0 192.168.10.0 0.0.0.0 255.255.255.192 U 0 0 0 br10 192.168.20.0 0.0.0.0 255.255.255.128 U 0 0 0 br20 192.168.22.0 0.0.0.0 255.255.255.128 U 0 0 0 br22 192.168.24.0 0.0.0.0 255.255.255.128 U 0 0 0 br24 192.168.90.0 0.0.0.0 255.255.255.128 U 0 0 0 br90 192.168.100.0 0.0.0.0 255.255.255.128 U 0 0 0 br100 192.168.102.0 10.255.253.2 255.255.255.192 UG 0 0 0 br4040 192.168.104.0 10.255.253.2 255.255.255.192 UG 0 0 0 br4040 192.168.108.0 0.0.0.0 255.255.255.128 U 0 0 0 br108 192.168.110.0 0.0.0.0 255.255.255.128 U 0 0 0 br110 192.168.254.10 192.168.110.31 255.255.255.255 UGH 0 0 0 br110 Note the last line in the table that shows the IP address from CiliumLoadBalancerIPPool and the next-hop IP address, which is one of the Kubernetes nodes.\nThings are shaping up nicely! 😎\nWorkload testing # Let me define a test deployment and service using the traefik/whoami container image.\n# whoami.yaml tee ./whoami.yaml \u0026lt;\u0026lt; EOF \u0026gt; /dev/null --- apiVersion: v1 kind: Namespace metadata: name: whoami --- apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: httproute-whoami namespace: whoami spec: parentRefs: - name: my-gateway namespace: infra-gateway sectionName: http rules: - backendRefs: - name: whoami port: 80 --- apiVersion: v1 kind: Service metadata: name: whoami namespace: whoami labels: app: whoami service: whoami spec: ports: - port: 80 name: http selector: app: whoami --- apiVersion: apps/v1 kind: Deployment metadata: name: whoami-http namespace: whoami spec: replicas: 3 selector: matchLabels: app: whoami template: metadata: labels: app: whoami spec: containers: - name: whoami image: traefik/whoami ports: - name: web containerPort: 80 EOF Deploy the workload by running kubectl apply -f ./whoami.yaml, then validate that the pods are running across all nodes by running: kubectl get pods -n default -o wide\n❯ kubectl get pods -n whoami -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES whoami-http-68965c9df8-4kqw4 1/1 Running 0 48s 10.244.1.209 dev-clus-core-cp03 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; whoami-http-68965c9df8-gzhjj 1/1 Running 0 48s 10.244.0.21 dev-clus-core-cp02 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; whoami-http-68965c9df8-xj4wl 1/1 Running 0 48s 10.244.3.248 dev-clus-core-cp01 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; From your workstation, let us get the IP address for the Gateway, and then try to curl the /api endpoint.\n❯ GATEWAY=$(kubectl get -n infra-gateway gateway my-gateway -o json | jq -r \u0026#39;.status.addresses[].value\u0026#39;) curl -s http://$GATEWAY/api | jq \u0026#39;.\u0026#39; { \u0026#34;hostname\u0026#34;: \u0026#34;whoami-http-68965c9df8-gzhjj\u0026#34;, \u0026#34;ip\u0026#34;: [ \u0026#34;127.0.0.1\u0026#34;, \u0026#34;::1\u0026#34;, \u0026#34;10.244.0.21\u0026#34;, \u0026#34;fe80::9482:c0ff:fecb:938a\u0026#34; ], \u0026#34;headers\u0026#34;: { \u0026#34;Accept\u0026#34;: [ \u0026#34;*/*\u0026#34; ], \u0026#34;User-Agent\u0026#34;: [ \u0026#34;curl/8.7.1\u0026#34; ], \u0026#34;X-Envoy-Internal\u0026#34;: [ \u0026#34;true\u0026#34; ], \u0026#34;X-Forwarded-For\u0026#34;: [ \u0026#34;192.168.20.93\u0026#34; ], \u0026#34;X-Forwarded-Proto\u0026#34;: [ \u0026#34;http\u0026#34; ], \u0026#34;X-Request-Id\u0026#34;: [ \u0026#34;2cffe316-36b5-42aa-87fe-48d01837cc38\u0026#34; ] }, \u0026#34;url\u0026#34;: \u0026#34;/api\u0026#34;, \u0026#34;host\u0026#34;: \u0026#34;192.168.254.10\u0026#34;, \u0026#34;method\u0026#34;: \u0026#34;GET\u0026#34;, \u0026#34;remoteAddr\u0026#34;: \u0026#34;10.244.3.205:43423\u0026#34; } Let us try a series of curl commands by running:\n❯ for i in $(seq 0 11); do curl -s http://$GATEWAY/api | jq \u0026#39;.hostname\u0026#39;; sleep 1; done \u0026#34;whoami-http-68965c9df8-4kqw4\u0026#34; \u0026#34;whoami-http-68965c9df8-4kqw4\u0026#34; \u0026#34;whoami-http-68965c9df8-xj4wl\u0026#34; \u0026#34;whoami-http-68965c9df8-gzhjj\u0026#34; \u0026#34;whoami-http-68965c9df8-xj4wl\u0026#34; \u0026#34;whoami-http-68965c9df8-4kqw4\u0026#34; \u0026#34;whoami-http-68965c9df8-xj4wl\u0026#34; \u0026#34;whoami-http-68965c9df8-xj4wl\u0026#34; \u0026#34;whoami-http-68965c9df8-gzhjj\u0026#34; \u0026#34;whoami-http-68965c9df8-gzhjj\u0026#34; \u0026#34;whoami-http-68965c9df8-gzhjj\u0026#34; \u0026#34;whoami-http-68965c9df8-4kqw4\u0026#34; Sweet! I can access all pods over the LoadBalancer IP address, which Cilium advertised into UniFi\u0026rsquo;s route table.\nLet me clean up, as I want to tweak this solution further.\nkubectl delete -f ./whoami.yaml kubectl delete -f ./infra/cilium/gateway.yaml Conclusion # In this article, I went over how to set up UniFi\u0026rsquo;s latest OS, which includes a UI for configuring BGP instead of hacking the router. I also showed how to enable and configure Cilium\u0026rsquo;s BGP and Gateway API features to complete the BGP configuration and create a simple ingress into a Kubernetes cluster. In my next article, I will explore setting up cert-manager and improving the Gateway configuration to create a reverse proxy using Cilium.\n","date":"2025-01","externalUrl":null,"permalink":"/posts/howto-setup-kubernetes-cilium-bgp-with-unifi-v4.1-router/","section":"Posts","summary":"In my previous article on building a Kubernetes cluster with Talos Linux, I used a Kubernetes Service of type NodePort to expose a workload to my homelab network. However, exposing workloads using NodePorts is not efficient or standard practice. In this article, I will document how I configured Cilium’s Gateway API as a basic reverse proxy and BGP Control Plane to inject routing paths into the routing table of a UniFi router for the reverse proxy IP address.\n","title":"Kubernetes BGP Connectivity with a UniFi router","type":"posts"},{"content":"","date":"2025-01","externalUrl":null,"permalink":"/tags/unifi/","section":"Tags","summary":"","title":"Unifi","type":"tags"},{"content":"So far in my container journey, I have used stand-alone hosts with Podman and Hashicorp Nomad (again backed by Podman) for container orchestration. While these endeavors worked, they were not the most popular option for managing a containerized workload cluster. Enter Kubernetes. Some months ago, I successfully deployed RKE2 with Rancher, but the solution was not stable. For example, during host reboots, Pods may not come back in a healthy state. Recently, I learned about Talos Linux and decided to try it. This article documents my effort to set up a Talos cluster in Proxmox virtual machines.\nSo, what is Talos Linux? This excerpt from https://www.talos.dev/ sums it up:\nTalos Linux is Linux designed for Kubernetes – secure, immutable, and minimal.\nSupports cloud platforms, bare metal, and virtualization platforms All system management is done via an API. No SSH, shell, or console Production ready: supports some of the largest Kubernetes clusters in the world Open source project from the team at Sidero Labs So here are some of my goals for a Kubernetes cluster:\nEasily deployable Kubernetes clusters on-premises using virtual machines or bare-metal Decommission my existing Nomad cluster Explore using passthrough GPUs A solution where I can learn more about the Kubernetes way of life Before I get started, predetermine the cluster\u0026rsquo;s virtual IP, name, and node IP addresses.\nCluster Name Cluster Virtual IP dev-clus-core 192.168.110.30 Name Type IP Address dev-clus-core-cp01 control-plane 192.168.110.31 dev-clus-core-cp02 control-plane 192.168.110.32 dev-clus-core-cp03 control-plane 192.168.110.33 Image Factory # Talos Linux has provided a way to customize ISO and disk images to your specifications using the Image Factory. Use your browser to go there and continue.\nChoose Cloud Server and click Next. Choosing the hardware type Select the latest version and click Next Selecting the version of Talos Linux Choose Nocloud. This option is required so that cloud-init can be leveraged to automate parts of the setup. Click Next Choose NoCloud Choose an appropriate hardware architecture (most likely amd64). Click Next Choose amd64 machine architecture OOn the Systems Extensions screen, I want to customize the disk image to include the necessary bits for Intel Arc GPUs (i915 and mei). I also want to include the qemu guest agent for the virtual machines. Search for Intel and qemu, select the following, and click Next: siderolabs/i915 siderolabs/mei siderolabs/qemu-guest-agent Choosing system extensions Click Next\nCopy the link for the Disk Image\nCopy the link for the disk image Proxmox # In the Proxmox environment, I need to download the disk image and create a template virtual machine for easy deployment. I am deploying three virtual machines for this initial test, each with the assigned control plane and worker roles.\nSSH into one of the Proxmox nodes and change the directory to a template volume. If you have not customized this, the default location is /var/lib/vz/template/iso cd /\u0026lt;template-volume-path\u0026gt;/template/iso/ Use curl to download the disk image that the Talos Factory generated curl -fLO https://factory.talos.dev/image/bbb84ab9bc2d8703ff7f0c46f04e20fee5e78d8c9af1cec7ce246f5b278dc0e5/v1.9.1/nocloud-amd64.raw.xz Uncompress the downloaded file unxz --decompress nocloud-amd64.raw.xz Create a virtual machine based on the image file the Image Factory generated. I also include the cloud-init \u0026ldquo;CD-Rom\u0026rdquo;. qm create 999999997 --name talos-1.9.1-template \\ --description \u0026#34;Template for Talos Linux v1.9.1\u0026#34; \\ --sockets 1 --cores 2 --cpu host \\ --memory 2048 --balloon 0 \\ --net0 virtio,bridge=vmbr0,tag=100 \\ --ostype l26 --agent 1,fstrim_cloned_disks=1 \\ --scsihw virtio-scsi-pci --boot order=scsi0 \\ --scsi0 ssd-pool:0,import-from=/\u0026lt;template-volume-path\u0026gt;/template/iso/nocloud-amd64.raw,ssd=1 \\ --scsi1 ssd-pool:10,ssd=1 \\ --ide2 ssd-pool:cloudinit \\ --serial0 socket --vga serial0 --citype nocloud Convert the virtual machine to a template qm template 999999997 Deploy three virtual machines based off this template qm clone 999999997 104 --full --name dev-clus-core-cp01 qm clone 999999997 105 --full --name dev-clus-core-cp02 qm clone 999999997 106 --full --name dev-clus-core-cp03 Configure the required CPU and memory for the workloads for each virtual machine for i in $(seq 104 106); do qm set $i --cores 4 --memory 16384; done Resize the disks for the workloads on each virtual machine for i in $(seq 104 106); do qm disk resize $i scsi0 20G; done for i in $(seq 104 106); do qm disk resize $i scsi1 50G; done Configure cloud-init and the network interface for each virtual machine. Here I am configuring static network settings for each virtual machine via cloud-init qm set 104 \\ --ipconfig0 ip=192.168.110.31/25,gw=192.168.110.1 \\ --ciupgrade 0 --net0 virtio,bridge=vmbr0,tag=110 \\ --nameserver \u0026#34;192.168.108.11 192.168.108.10\u0026#34; \\ --searchdomain lab.howto.engineer qm set 105 \\ --ipconfig0 ip=192.168.110.32/25,gw=192.168.110.1 \\ --ciupgrade 0 --net0 virtio,bridge=vmbr0,tag=110 \\ --nameserver \u0026#34;192.168.108.11 192.168.108.10\u0026#34; \\ --searchdomain lab.howto.engineer qm set 106 \\ --ipconfig0 ip=192.168.110.33/25,gw=192.168.110.1 \\ --ciupgrade 0 --net0 virtio,bridge=vmbr0,tag=110 \\ --nameserver \u0026#34;192.168.108.11 192.168.108.10\u0026#34; \\ --searchdomain lab.howto.engineer [Optional] Add the nodes to an HA group. Otherwise, power them on manually ha-manager add 104 --group ha-global-group ha-manager add 105 --group ha-global-group ha-manager add 106 --group ha-global-group Talos Linux configuration # Create DNS records for the for the virtual IP and the nodes\nIf not already installed on MacOS, install kubectl\nVERSION=\u0026#34;v1.32.0\u0026#34; curl -LO \u0026#34;https://dl.k8s.io/release/${VERSION}/bin/darwin/amd64/kubectl\u0026#34; chmod +x ./kubectl sudo mv ./kubectl /usr/local/bin/ sudo chown root: /usr/local/bin/kubectl On MacOS, install talosctl brew install siderolabs/tap/talosctl Create a temporary directory to store configuration files mkdir -p talos_config/setup \u0026amp;\u0026amp; cd talos_config Create patch files to customize Talos Linux and Kubernetes # Disable the predictable network interface names tee ./setup/stable-network-interfaces.yaml \u0026lt;\u0026lt; EOF \u0026gt; /dev/null --- machine: install: extraKernelArgs: - net.ifnames=0 EOF # Set the Talos virtual IP tee ./setup/kube-api-vip.yaml \u0026lt;\u0026lt; EOF \u0026gt; /dev/null --- machine: network: interfaces: - interface: eth0 vip: ip: 192.168.110.30 EOF # Define storage tee ./setup/storage.yaml \u0026lt;\u0026lt; EOF \u0026gt; /dev/null --- machine: install: disk: /dev/sda disks: - device: /dev/sdb partitions: - mountpoint: /var/local EOF # Define NTP sources tee ./setup/ntp-sources.yaml \u0026lt;\u0026lt; EOF \u0026gt; /dev/null --- machine: time: disabled: false servers: - 172.16.2.1 - us.pool.ntp.org bootTimeout: 2m0s EOF # Define a Metrics Server tee ./setup/enable-metrics.yaml \u0026lt;\u0026lt; EOF \u0026gt; /dev/null machine: kubelet: extraArgs: rotate-server-certificates: true cluster: extraManifests: - https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver/main/deploy/standalone-install.yaml - https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml EOF # Allow scheduling of pods on control plane nodes tee ./setup/pod-scheduling.yaml \u0026lt;\u0026lt; EOF \u0026gt; /dev/null --- cluster: allowSchedulingOnControlPlanes: true EOF # Disable kube-proxy and the default Flannel CNI tee ./setup/disable-cni.yaml \u0026lt;\u0026lt; EOF \u0026gt; /dev/null --- cluster: network: cni: name: none proxy: disabled: true EOF Generate a secrets bundle. It is an optional step if secrets must be separated from the configuration in case the configuration is in source control. talosctl gen secrets --output-file ./secrets.yaml Generate a patched Talos configuration. Three files are created by talosctl gen config: controlplane.yaml, worker.yaml, and talosconfig. The first two will be applied to control-plane and worker nodes, respectively. In this tutorial, I only use controlplane.yaml as both roles are installed per node. talosconfig contains metadata and secrets for managing a Talos Linux cluster. talosctl gen config dev-clus-core https://192.168.110.30:6443 \\ --with-secrets ./secrets.yaml \\ --kubernetes-version \u0026#34;1.32.0\u0026#34; \\ --config-patch @setup/storage.yaml \\ --config-patch @setup/enable-metrics.yaml \\ --config-patch @setup/stable-network-interfaces.yaml \\ --config-patch @setup/pod-scheduling.yaml \\ --config-patch @setup/ntp-sources.yaml \\ --config-patch @setup/kube-api-vip.yaml \\ --config-patch @setup/disable-cni.yaml \\ --output . Move the Talos configuration file to its default directory if this is the first Talos cluster. To merge configurations, opt to use talosctl config merge ./talosconfig [ ! -d ~/.config/talos ] \u0026amp;\u0026amp; mkdir -p ~/.config/talos || : mv -i ./talosconfig ~/.config/talos/config.yaml Apply the patched configuration (controlplane.yaml) to each node talosctl apply-config \\ --file ./controlplane.yaml \\ --nodes 192.168.110.31 --insecure talosctl apply-config \\ --file ./controlplane.yaml \\ --nodes 192.168.110.32 --insecure talosctl apply-config \\ --file ./controlplane.yaml \\ --nodes 192.168.110.33 --insecure Configure the Talos configuration context with endpoints that refer to each node in the cluster talosctl config endpoint 192.168.110.31 192.168.110.32 192.168.110.33 Review Talos\u0026rsquo; cluster members talosctl get members --nodes 192.168.110.31 Bootstrap Kubernetes talosctl bootstrap --nodes 192.168.110.31 Get Kubernetes\u0026rsquo; kubeconfig from the Talos cluster and merge it to ~/.kube/config talosctl kubeconfig --nodes 192.168.110.31 Use kubectl to review the state of the cluster\u0026rsquo;s nodes and pods. kubectl get nodes -o wide \u0026amp;\u0026amp; \\ kubectl get pods -A Note Notice that the nodes and a few pods are not in a Ready/Running state. The cluster has not defined a CNI, which will cause the cluster to be in a not-ready state.\nInstall the Cilium CNI helm repo add cilium https://helm.cilium.io/ helm repo update helm install cilium cilium/cilium --version 1.16.5 \\ --namespace kube-system \\ --set ipam.mode=kubernetes \\ --set=kubeProxyReplacement=true \\ --set=securityContext.capabilities.ciliumAgent=\u0026#34;{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}\u0026#34; \\ --set=securityContext.capabilities.cleanCiliumState=\u0026#34;{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}\u0026#34; \\ --set=cgroup.autoMount.enabled=false \\ --set=cgroup.hostRoot=/sys/fs/cgroup \\ --set=k8sServiceHost=localhost \\ --set=k8sServicePort=7445 Restart unmanaged pods kubectl get pods --all-namespaces -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,HOSTNETWORK:.spec.hostNetwork \\ --no-headers=true | grep \u0026#39;\u0026lt;none\u0026gt;\u0026#39; | awk \u0026#39;{print \u0026#34;-n \u0026#34;$1\u0026#34; \u0026#34;$2}\u0026#39; | xargs -L 1 kubectl delete pod Give the cluster a few minutes to restart pods and get into a Running state Reviewing Kubernetes node and pod statuses Take a look at Talos\u0026rsquo; dashboard to review the state of the cluster by running: talosctl dashboard --nodes \u0026quot;192.168.110.31,192.168.110.32,192.168.110.33\u0026quot; Reviewing Talos Linux\u0026rsquo;s dashboard Kubernetes deployment test # Create a test deployment of nginx to the cluster and expose it using a NodePort service kubectl create deployment nginx-test --image=nginx:latest kubectl expose deployment nginx-test --type NodePort --port 80 Get the node the pod is running on kubectl get pod -o=custom-columns=NAME:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName -n default Get the port the service is available on kubectl get service nginx-test -o=custom-columns=NAME:.metadata.name,PORT:.spec.ports[].nodePort Point a web browser to the hostname:port to validate that the Pod is exposed and working Viewing the default nginx start page Clean up the service and deployment\nkubectl delete service/nginx-test kubectl delete deployment/nginx-test Conclusion # All in all, this was easy to configure and deploy for my needs. Documentation was sufficient, and the Slack community was helpful. So far, Talos Linux has been ticking the boxes. There is one improvement that I wish was there with Proxmox, though. Technically, I can include the controlplane.yaml or worker.yaml in the cloud-init configuration, but the good folks at Proxmox have not provided a clean way to modify the file via a UI or API. One must log into the shell of a node to modify the file, and at this stage of my discovery, it is easier to make edits and apply changes outside of Proxmox.\nIn a future article, I will take another step in my Kubernetes journey to configure Cilium to work with my existing lab network.\n","date":"2024-12","externalUrl":null,"permalink":"/posts/getting-started-with-talos-linux-on-proxmox/","section":"Posts","summary":"So far in my container journey, I have used stand-alone hosts with Podman and Hashicorp Nomad (again backed by Podman) for container orchestration. While these endeavors worked, they were not the most popular option for managing a containerized workload cluster. Enter Kubernetes. Some months ago, I successfully deployed RKE2 with Rancher, but the solution was not stable. For example, during host reboots, Pods may not come back in a healthy state. Recently, I learned about Talos Linux and decided to try it. This article documents my effort to set up a Talos cluster in Proxmox virtual machines.\n","title":"Getting started with Talos Linux on Proxmox","type":"posts"},{"content":"","date":"2024-12","externalUrl":null,"permalink":"/tags/kubernetes/","section":"Tags","summary":"","title":"Kubernetes","type":"tags"},{"content":"","date":"2024-12","externalUrl":null,"permalink":"/tags/talos/","section":"Tags","summary":"","title":"Talos","type":"tags"},{"content":"","date":"2024-11","externalUrl":null,"permalink":"/tags/authentication/","section":"Tags","summary":"","title":"Authentication","type":"tags"},{"content":"","date":"2024-11","externalUrl":null,"permalink":"/categories/cloud/","section":"Categories","summary":"","title":"Cloud","type":"categories"},{"content":"","date":"2024-11","externalUrl":null,"permalink":"/tags/gitlab/","section":"Tags","summary":"","title":"Gitlab","type":"tags"},{"content":"Using JSON keys to authenticate with Google Cloud is highly frowned upon. Unless you have no other option, Google Cloud provides a more secure means of authenticating externally executed code. My use case is for authentication in GitLab pipelines so that I can automate tasks. Think Terraform jobs or updating the files for a website stored in a Google Cloud storage bucket. I will use Google Cloud\u0026rsquo;s Workload Identity Federation solution and the OIDC (Open ID Connect) protocol in this solution.\nTo get started, here are a few requirements:\nA GitLab project that will contain the pipeline code. A Google Cloud project with a service to grant access to. I refer to this as the service project below. Create a Google Cloud project specifically for Workload Identity gcloud projects create workload-id --no-enable-cloud-apis gcloud config set project workload-id Note Validate that you have the roles/iam.workloadIdentityPoolAdmin role assigned to yourself on the project at a minimum or the roles/owner role if you are in a development environment.\nEnable the required APIs for Workload Identity gcloud services enable \\ cloudresourcemanager.googleapis.com \\ iam.googleapis.com \\ iamcredentials.googleapis.com \\ sts.googleapis.com \\ --project=\u0026#34;workload-id\u0026#34; Define a few variables. WIP_PROJECT_ID=$(gcloud projects describe $(gcloud config get-value core/project) --format=\u0026#34;get(projectId)\u0026#34;) WIP_PROJECT_NUMBER=$(gcloud projects describe $(gcloud config get-value core/project) --format=\u0026#34;get(projectNumber)\u0026#34;) GITLAB_GROUP=\u0026#34;originaltrini0-codes\u0026#34; GITLAB_PROJECT_NUMBER=\u0026#34;64700261\u0026#34; WIP_NAME=\u0026#34;test-wip\u0026#34; WIP_PROVIDER_NAME=\u0026#34;test-wip-provider\u0026#34; SERVICE_PROJECT_ID=\u0026#34;test-compute-443418\u0026#34; Note I organize my GitLab projects in groups. The parent group can be found in the URL: https://gitlab.com/your-group-name\nLook for the GitLab project number under General -\u0026gt; Settings of a GitLab project\nCreate a Workload Identity Pool gcloud iam workload-identity-pools create $WIP_NAME \\ --location=\u0026#34;global\u0026#34; \\ --display-name=\u0026#34;Test WIP\u0026#34; \\ --description=\u0026#34;Test Workload Identity Pool\u0026#34; \\ --project=\u0026#34;${WIP_PROJECT_ID}\u0026#34; Create a Workload Identity Provider for GitLab gcloud iam workload-identity-pools providers create-oidc $WIP_PROVIDER_NAME \\ --location=\u0026#34;global\u0026#34; \\ --workload-identity-pool=\u0026#34;${WIP_NAME}\u0026#34; \\ --description=\u0026#34;Test Workload Identity OIDC Provider\u0026#34; \\ --display-name=\u0026#34;Test Workload Identity Provider\u0026#34; \\ --issuer-uri=\u0026#34;https://gitlab.com/\u0026#34; \\ --attribute-mapping=\u0026#34;attribute.guest_access=assertion.guest_access,\\ attribute.planner_access=assertion.planner_access,\\ attribute.reporter_access=assertion.reporter_access,\\ attribute.developer_access=assertion.developer_access,\\ attribute.maintainer_access=assertion.maintainer_access,\\ attribute.owner_access=assertion.owner_access,\\ attribute.namespace_id=assertion.namespace_id,\\ attribute.namespace_path=assertion.namespace_path,\\ attribute.project_id=assertion.project_id,\\ attribute.project_path=assertion.project_path,\\ attribute.user_id=assertion.user_id,\\ attribute.user_login=assertion.user_login,\\ attribute.user_email=assertion.user_email,\\ attribute.user_access_level=assertion.user_access_level,\\ google.subject=assertion.sub\u0026#34; \\ --attribute-condition=\u0026#34;assertion.namespace_id==\\\u0026#34;${GITLAB_GROUP}\\\u0026#34;\u0026#34; \\ --project=\u0026#34;${WIP_PROJECT_ID}\u0026#34; Note Use the --attribute-condition to fine tune the parameters in the GitLab assertion to match against. [link]\nCreate a service account for impersonation in the service project gcloud iam service-accounts create test-svc-acct \\ --display-name=\u0026#34;Test Service Account\u0026#34; \\ --project=\u0026#34;${SERVICE_PROJECT_ID}\u0026#34; Create a storage bucket in the service project and upload test files to it gcloud storage buckets create gs://gms-bucket-2024 \\ --project=\u0026#34;${SERVICE_PROJECT_ID}\u0026#34; gcloud storage cp ~/Downloads/*.jpg gs://gms-bucket-2024 Get the service project\u0026rsquo;s service account email address and grant it the Workload Identity User role SERVICE_ACCOUNT_EMAIL=$(gcloud iam service-accounts list --project=\u0026#34;${SERVICE_PROJECT_ID}\u0026#34; --filter=\u0026#34;email~test-svc-acct\u0026#34; --format=\u0026#34;get(email)\u0026#34;) gcloud iam service-accounts add-iam-policy-binding $SERVICE_ACCOUNT_EMAIL \\ --role=roles/iam.workloadIdentityUser \\ --member=\u0026#34;principalSet://iam.googleapis.com/projects/${WIP_PROJECT_NUMBER}/locations/global/workloadIdentityPools/${WIP_NAME}/attribute.project_id/${GITLAB_PROJECT_NUMBER}\u0026#34; \\ --project=\u0026#34;${SERVICE_PROJECT_ID}\u0026#34; Grant the service account the roles/storage.objectAdmin role to the storage bucket gcloud storage buckets add-iam-policy-binding gs://gms-bucket-2024 \\ --member=serviceAccount:$SERVICE_ACCOUNT_EMAIL \\ --role=roles/storage.objectAdmin \\ --project=\u0026#34;${SERVICE_PROJECT_ID}\u0026#34; In your GitLab project, commit a .gitlab-ci.yml file with the following content stages: - auth auth: variables: GOOGLE_CLOUD_PROJECT: test-compute-443418 GOOGLE_SERVICE_ACCOUNT_EMAIL: test-svc-acct@test-compute-443418.iam.gserviceaccount.com GOOGLE_WORKLOAD_IDENTITY_PROVIDER: projects/722905479622/locations/global/workloadIdentityPools/test-wip/providers/test-wip-provider image: docker.io/google/cloud-sdk:slim id_tokens: GITLAB_OIDC_TOKEN: aud: https://iam.googleapis.com/projects/722905479622/locations/global/workloadIdentityPools/test-wip/providers/test-wip-provider script: - mkdir -p _google_auth - echo \u0026#34;${GITLAB_OIDC_TOKEN}\u0026#34; \u0026gt; $CI_PROJECT_DIR/_google_auth/.ci_job_jwt_file - gcloud iam workload-identity-pools create-cred-config ${GOOGLE_WORKLOAD_IDENTITY_PROVIDER} --service-account=\u0026#34;${GOOGLE_SERVICE_ACCOUNT_EMAIL}\u0026#34; --service-account-token-lifetime-seconds=600 --output-file=$CI_PROJECT_DIR/_google_auth/.gcp_temp_cred.json --credential-source-file=$CI_PROJECT_DIR/_google_auth/.ci_job_jwt_file - gcloud config set project test-compute-443418 - gcloud auth login --cred-file=$CI_PROJECT_DIR/_google_auth/.gcp_temp_cred.json - gcloud storage ls gs://gms-bucket-2024 stage: auth when: manual Manually trigger the job\nReview the job\u0026rsquo;s progress. The results show that the GitLab runner can authenticate and list the files in the storage bucket Gitlab pipeline results Go to Logs Explorer for the service project, and run the following query\nprotoPayload.methodName=\u0026#34;GenerateAccessToken\u0026#34; AND protoPayload.authorizationInfo.granted=\u0026#34;true\u0026#34; You should find the corresponding log entry similar to this screenshot where the GitLab runner authenticates to Google Cloud Authentication logs in Google Cloud Logging Conclusion # In this article, I demonstrate how to enable and configure a Workload Identity Pool and Provider that uses GitLab\u0026rsquo;s assertion data to grant access to Google Cloud. For more information about Workload Identity Federation, visit the documentation page.\n","date":"2024-11","externalUrl":null,"permalink":"/posts/how-to-configure-google-cloud-workload-identities-with-gitlab/","section":"Posts","summary":"Using JSON keys to authenticate with Google Cloud is highly frowned upon. Unless you have no other option, Google Cloud provides a more secure means of authenticating externally executed code. My use case is for authentication in GitLab pipelines so that I can automate tasks. Think Terraform jobs or updating the files for a website stored in a Google Cloud storage bucket. I will use Google Cloud’s Workload Identity Federation solution and the OIDC (Open ID Connect) protocol in this solution.\n","title":"Google Cloud Workload Identities with GitLab","type":"posts"},{"content":"","date":"2024-11","externalUrl":null,"permalink":"/tags/google-cloud/","section":"Tags","summary":"","title":"Google-Cloud","type":"tags"},{"content":"","date":"2024-11","externalUrl":null,"permalink":"/tags/ci/cd/","section":"Tags","summary":"","title":"Ci/Cd","type":"tags"},{"content":"I want to get my hands dirty with CI/CD. After looking around at cloud-hosted options such as Google Cloud Build and Azure DevOps/Pipelines, I decided to keep this process local by leveraging self-managed GitLab CI/CD pipelines. To run a GitLab pipeline, you need only a special configuration file, .gitlab-ci.yml, at the root of your GitLab project/repository and at least one or more compute resources to execute jobs. In this article, I will discuss how I set up a GitLab runner using Podman.\nA GitLab Runner, in its simplest form, is a compute resource capable of executing jobs defined in a pipeline and can be installed on Windows, FreeBSD, and Linux. Once installed, GitLab registers the compute endpoints and can send commands to the compute endpoint using the instructions laid out in the pipeline. There are a few other ways to run the GitLab Runner application. You can also use Docker or Kubernetes to execute GitLab runner pipelines. I chose this route because I can use Podman as a drop-in replacement for Docker, and GitLab supports it. I chose this method instead of installing static applications, which could lead to operating system bloat and the risk of inconsistent results because job containers are ephemeral. The goal is to keep a pristine operating system and use throw-away containers to execute jobs.\nInstallation # For my needs which is a bit overkill, I am deploying two virtual machines running Fedora, each with the following specs:\n4 vCPU 4GB RAM 20GB HDD Install the latest version of GitLab Runner:\nsudo dnf install --assumeyes \\ https://s3.dualstack.us-east-1.amazonaws.com/gitlab-runner-downloads/latest/rpm/gitlab-runner_amd64.rpm Registration # Now that I have installed the GitLab Runner, I can register the compute nodes in GitLab.\nLog into GitLab\nGo to your parent GitLab root group\nOn the left pane, go to Settings -\u0026gt; CI/CD\nExpand Runners\nEnsure that instance runners are enabled for the group Enabling GitLab instance runners Now go to Build -\u0026gt; Runners\nClick the New group runner button Creating a new group runner On the New group runner page, ensure that Run untagged jobs is enabled, and for Runner description, enter the name of the server Group runner options Click the Create runner button\nOn the next page, you will get a command to run on your runner virtual machine. Copy it. Group runner registration command Run the generated command from the previous step with sudo privileges\nsudo gitlab-runner register --url https://gitlab.com --token \u0026lt;generated-token\u0026gt; Accept the defaults until the question about which executor to use\nEnter docker\nNote Choosing Docker here does not affect how I will be using Podman\nFor the default Docker image question, enter alpine:latest\nIf all goes well, the wizard will complete, showing the location of the configuration file /etc/gitlab-runner/config.toml\nValidate that the service is running\nsystemctl status gitlab-runner Go back to the GitLab portal. At the bottom, there is a button called View runners. Click it to view your registered runners! Viewing registered runners Podman configuration # I will configure the Gitlab runner application to use Podman in this section.\nAllow runners to create a container network by enabling the FF_NETWORK_PER_BUILD feature flag sudo tee --append /etc/gitlab-runner/config.toml \u0026lt;\u0026lt; EOF \u0026gt; /dev/null [runners.feature_flags] FF_NETWORK_PER_BUILD = true EOF TThe GitLab runner application created a gitlab-runner user account. Add the user account to the /etc/subuid and /etc/subgid files x=$(tail -n 1 /etc/subuid | cut -d \u0026#39;:\u0026#39; -f 2) sudo usermod --add-subuids $((x+65536))-$((x+(65536*2)-1)) gitlab-runner x=$(tail -n 1 /etc/subgid | cut -d \u0026#39;:\u0026#39; -f 2) sudo usermod --add-subgids $((x+65536))-$((x+(65536*2)-1)) gitlab-runner I must log in as the gitlab-runner user account to continue the configuration.\nTemporarily configure a password for the gitlab-runner user account sudo passwd gitlab-runner Temporarily enable SSH password login sudo sed -i \u0026#39;s/PasswordAuthentication no/PasswordAuthentication yes/\u0026#39; /etc/ssh/sshd_config.d/50-cloud-init.conf sudo systemctl reload sshd Log into the gitlab-runner account with the temporary password ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no gitlab-runner@localhost As the gitlab-runner account, enable and start the Podman socket. Then, check the status of the service systemctl --user --now enable podman.socket systemctl status --user podman.socket Note the Listen line from the previous step\u0026rsquo;s systemctl status output. We need the path to the socket file. For example, it could look similar: Listen: /run/user/992/podman/podman.sock (Stream)\nLogout of the gitlab-runner shell logout Remove the temporary password on the gitlab-runner user account sudo passwd -ld gitlab-runner Disable SSH password authentication sudo sed -i \u0026#39;s/PasswordAuthentication yes/PasswordAuthentication no/\u0026#39; /etc/ssh/sshd_config.d/50-cloud-init.conf sudo systemctl reload sshd Remove localhost from SSH\u0026rsquo;s known_hosts ssh-keygen -R localhost Configure the gitlab-runner shell to linger sudo loginctl enable-linger gitlab-runner Now I will add the path to Podman\u0026rsquo;s socket file to /etc/gitlab-runner/config.toml sudo sed -i \u0026#39;/\\[runners.docker\\]/a\\ host = \u0026#34;unix:\\/\\/\\/run\\/user\\/992\\/podman\\/podman.sock\u0026#34;\u0026#39; /etc/gitlab-runner/config.toml Note Be careful and properly escape the string when using sed.\nRestart the gitlab-runner service sudo systemctl restart gitlab-runner Testing the solution # Create a test GitLab project\nIn the newly created project, turn off GitLab\u0026rsquo;s hosted instance runners by navigating to Settings -\u0026gt; CI/CD -\u0026gt; Runners Turning off GitLab hosted runners Commit a .gitlab-ci.yml file to the root of the project with the following content\nbuild-job: stage: build script: - echo \u0026#34;Hello, $GITLAB_USER_LOGIN!\u0026#34; test-job1: stage: test script: - echo \u0026#34;This job tests something\u0026#34; test-job2: stage: test script: - echo \u0026#34;This job tests something, but takes more time than test-job1.\u0026#34; - echo \u0026#34;After the echo commands complete, it runs the sleep command for 20 seconds\u0026#34; - echo \u0026#34;which simulates a test that runs 20 seconds longer than test-job1\u0026#34; - sleep 20 deploy-prod: stage: deploy script: - echo \u0026#34;This job deploys something from the $CI_COMMIT_BRANCH branch.\u0026#34; environment: production Once the commit has gone through, go to Build -\u0026gt; Pipelines to get a visual overview of the steps and statuses Viewing job statuses Drill into one of the steps to see its execution progress. You should see something similar to this if all goes well Viewing test-job1 Viewing test-job2 Conclusion # With all the steps completed, you will have a base solution that allows you to programmatically execute pipelines to do almost anything you can think of. You can define custom container images in the .gitlab-ci.yml file, and shell scripting will create new opportunities to automate tasks! To learn more about the .gitlab-ci.yml configuration options, head to this page.\n","date":"2024-11","externalUrl":null,"permalink":"/posts/how-to-setup-self-managed-podman-gitlab-runner/","section":"Posts","summary":"I want to get my hands dirty with CI/CD. After looking around at cloud-hosted options such as Google Cloud Build and Azure DevOps/Pipelines, I decided to keep this process local by leveraging self-managed GitLab CI/CD pipelines. To run a GitLab pipeline, you need only a special configuration file, .gitlab-ci.yml, at the root of your GitLab project/repository and at least one or more compute resources to execute jobs. In this article, I will discuss how I set up a GitLab runner using Podman.\n","title":"How to setup a self-managed Podman Gitlab Runner","type":"posts"},{"content":"","date":"2024-11","externalUrl":null,"permalink":"/tags/podman/","section":"Tags","summary":"","title":"Podman","type":"tags"},{"content":"In my previous article on Google Cloud federation and account provisioning with Microsoft Entra ID, I showed how to get started to configure it. This article constitutes the second part, utilizing SAML authentication to complete the solution. Once you complete the steps here, you will have a secure means of logging into Google Cloud with your Entra ID account.\nTo complete this process, you will need to have completed the first portion as documented here.\nCreate a Google Admin SAML profile # In Google Admin, go to Security -\u0026gt; Authentication -\u0026gt; SSO with third-party\nClick \u0026ldquo;Add SAML Profile\u0026rdquo; Add a SAML profile Give it a name and click Save. You will fill in the other fields later\nThe page flow will generate SP (Service Provider) URLs for you. These will be needed when creating an Entra ID SSO application\nCreate a Microsoft Entra ID SSO application # In Microsoft Entra ID, create another \u0026ldquo;Google Cloud/G Suite Connector by Microsoft\u0026rdquo; application\nName it \u0026ldquo;Google Cloud SSO\u0026rdquo; Create an Entra ID SSO application After its creation, go to Manage -\u0026gt; Properties\nEnsure that both options are set to \u0026ldquo;Yes\u0026rdquo; for \u0026ldquo;Enabled for users to sign-in\u0026rdquo; and \u0026ldquo;Assignment required\u0026rdquo;\nClick Save\nGo to Manage -\u0026gt; Users and groups\nClick Add user/group to assign a user or group\nNote Since I\u0026rsquo;m using the free Microsoft Entra ID, I cannot add groups\nGo to Manage -\u0026gt; Single sign-on\nClick the \u0026ldquo;SAML\u0026rdquo; card Entra ID SAML card In the resulting page, click Edit on step 1 Entra ID SAML configuration In the \u0026ldquo;Identifier (Entity ID)\u0026rdquo; section, click the \u0026ldquo;Add Identifier\u0026rdquo; link\nPaste the \u0026ldquo;Entity ID\u0026rdquo; URL from the SSO Profile in Google Admin here and click the Default checkbox\nRemove any other entries here resulting with just the single entry Entra ID Identifier In the \u0026ldquo;Reply URL (Assertion Consumer Service URL)\u0026rdquo; section, click the \u0026ldquo;Add reply URL\u0026rdquo; link\nPaste the \u0026ldquo;ACS URL\u0026rdquo; from the SSO Profile in Google Admin here\nFor the Sign on URL, paste the following, replacing PRIMARY_DOMAIN with your FQDN: https://www.google.com/a/PRIMARY_DOMAIN/ServiceLogin?continue=https://console.cloud.google.com/\nClick Save\nClose the \u0026ldquo;Basic SAML Configuration\u0026rdquo; pane\nIn step 2, click Edit\nSince I am using the Universal Principal Name (UPN) as the identifier, please remove all additional claims and retain only the required claim configured for UPN Entra ID Attributes Make a note of the Base64 token signing certificate in step 3\nMake a note of the following URLs in step 4 \u0026ldquo;Login URL\u0026rdquo; \u0026ldquo;Microsoft Entra Identifier\u0026rdquo;\nComplete the Google Admin SSO profile # In the IDP details section from step 4 in the Microsoft Entra ID SSO application, paste the \u0026ldquo;Microsoft Entra Identifier\u0026rdquo; URL into the IDP entity ID field\nAlso paste in \u0026ldquo;Login URL\u0026rdquo; URL into the \u0026ldquo;Sign-in page URL\u0026rdquo; field\nPaste the following into the \u0026ldquo;Sign-out page URL\u0026rdquo; field: https://login.microsoftonline.com/common/wsfederation?wa=wsignout1.0\nPPaste the following into the \u0026ldquo;Change password URL\u0026rdquo; field: https://account.activedirectory.windowsazure.com/changepassword.aspx\nUpload the Base64 token signing certificate in step 3 in the Microsoft Entra ID SSO application to the Verification certificate\nClick Save\nThe result should look similar to this: Google Cloud Identity IdP details Assigning the SSO profile # In Google Admin, go to Security -\u0026gt; Authentication -\u0026gt; SSO with third-party IdP\nIn the \u0026ldquo;Manage SSO profile assignments\u0026rdquo; section, click the \u0026ldquo;Get Started\u0026rdquo; link\nIn the left pane, select the \u0026ldquo;Users\u0026rdquo; OU\nIn the right pane, select the \u0026ldquo;Another SSO profile\u0026rdquo; radio button\nIn the SSO profile dropdown menu, choose the SSO profile that you created earlier\nYour result should look similar to this: Google Identity SSO Profile assignment Click Override\nClick on the Automation OU\nChoose None for SSO profile assignment\nClick Override\nTesting single-sign on # With an incognito browser window, navigate to https://console.cloud.google.com\nEnter the username of an account that the Microsoft Entra ID automation has provisioned\nYou should get redirected to log into Microsoft Entra ID\nIf your credentials are correct, you will be redirected back to Google Cloud console\nAgree to the terms of service and click \u0026ldquo;Agree and continue\u0026rdquo; Google Cloud login flow Conclusion # And there you have it. If all goes well, this overall solution gives you the following:\nThe ability to use one account for both Microsoft/Azure and Google ecosystems The basic groundwork to expand into Entra ID\u0026rsquo;s conditional access The foundation to manage access to Google Cloud resources Have any questions? Feel free to reach out to me.\nThanks! \u0026#x1f44b;\n","date":"2024-09","externalUrl":null,"permalink":"/posts/how-to-federate-google-cloud-entra-id-part2/","section":"Posts","summary":"In my previous article on Google Cloud federation and account provisioning with Microsoft Entra ID, I showed how to get started to configure it. This article constitutes the second part, utilizing SAML authentication to complete the solution. Once you complete the steps here, you will have a secure means of logging into Google Cloud with your Entra ID account.\n","title":"Google Cloud federation with Microsoft Entra ID - Part 2","type":"posts"},{"content":"","date":"2024-09","externalUrl":null,"permalink":"/tags/microsoft-entra-id/","section":"Tags","summary":"","title":"Microsoft-Entra-Id","type":"tags"},{"content":"I wanted to use a single account to log into my Azure and Google Cloud environments and automatically provision \u0026ldquo;source of truth\u0026rdquo; accounts from Entra ID to Google Cloud Identity. This article will explain how I configured account provisioning of the identity federation solution between Microsoft Entra ID and Google Cloud Identity.\nPrerequisite # For this excercise, here are the basic items that I used for this solution\nAn internet domain name. For example: domain.tld\nA Microsoft 365 business account using your domain name\nA free Google Cloud Identity account\nTo create a Cloud Identity account, go to the Google Cloud Identity sign up page and follow the setup instructions for a free account.\nAfter creating your account, you must verify that you own the domain. The sign-up form will generate a unique code for your DNS record during the sign-up process. To complete the verification, create a TXT record using this code in your domain registrar or DNS portal, and follow the DNS verification steps to finish the process.\nGoogle admin steps # Log into the Google Admin console with your credentials used when signing up for Google Cloud Identity\nCreate a break glass account under Directory -\u0026gt; Users for emergencies. Do save the credentials in a Password manager. It can be used to log into the Google Admin console if the federation solution is not working and access is needed. Creating an account in Google Admin Note When setting the password for this account, ensure that it is a complex one and validate that \u0026ldquo;Ask user to change their password when they sign in\u0026rdquo; is unchecked\nCreate an OU for the Microsoft Entra ID automation account at Directory -\u0026gt; Organizational Units. Highlight the organization root and click the \u0026ldquo;+\u0026rdquo; button Creating an OU in Google Admin Click Create\nGo to Directory -\u0026gt; Users and create a new account that Microsoft Entra ID will use Entra ID account details Grant permissions for the automation account via a custom role. Navigate to Account -\u0026gt; Admin Roles and click on \u0026ldquo;Create new role\u0026rdquo; Creating an admin role Provide a name and description. Click Continue Google Admin role name and description Grant Organizational Units -\u0026gt; Read, Users and Groups. Click Continue Google Admin role rights Click on Create Role\nNow to assign accounts, click on Assign members Assigning an account to a role Search for the account and click on Assign Role Assigning an account to a role Create another OU for Microsoft Entra ID provisioned accounts Creating an OU in Google Admin Microsoft Entra ID Provisioning steps # In the Azure Portal, go to Microsoft Entra -\u0026gt; Enterprise applications\nClick on New application Creating a Microsoft Entra ID application Search for \u0026ldquo;Google Cloud\u0026rdquo; and click on the \u0026ldquo;Google Cloud / G Suite Connector by Microsoft\u0026rdquo; app Entra ID built-in applications Rename the app to something meaningful and click Create Google Cloud Entra ID application On the left, go to Manage -\u0026gt; Properties\nSelect No for:\n\u0026ldquo;Enabled for users to sign-in\u0026rdquo;\n\u0026ldquo;Assignment required\u0026rdquo;\n\u0026ldquo;Visible to users\u0026rdquo; Entra ID application configuration Click Save towards the top\nGo to Manage -\u0026gt; Provisioning and click \u0026ldquo;Get Started\u0026rdquo;\nChange \u0026ldquo;Provisioning Mode\u0026rdquo; to Automatic\nUnder Admin Credentials, click Authorize\nEnter the username/password for the provisioning account that you created earlier in Google Admin\nClick the \u0026ldquo;I understand\u0026rdquo; button to the new account agreement\nChoose \u0026ldquo;Select All\u0026rdquo; Choosing what Entra ID can access Click Continue\nClick the \u0026ldquo;Test Connection\u0026rdquo; button\nValidate the test completes successfully Confirmation Click Save\nUser and Group provisioning # In the Microsoft Entra ID provisioning app, go to Manage -\u0026gt; Provisioning\nExpand Mappings\nClick the \u0026ldquo;Provision Microsoft Entra ID Users\u0026rdquo; link For the following attributes, click the Edit button and set the field \u0026ldquo;Default value if null (optional)\u0026rdquo; to an underscore:\n\u0026ldquo;name.familyName\u0026rdquo; \u0026ldquo;name.givenName\u0026rdquo; Configure Entra ID user mappings Click OK\nClick Save at the top and confirm the changes\nGo back to the Provisioning -\u0026gt; Mapping and click the \u0026ldquo;Provision Microsoft Entra ID Groups\u0026rdquo; link\nClick Edit for the \u0026ldquo;email\u0026rdquo; attribute and change \u0026ldquo;GROUPS_DOMAIN\u0026rdquo; match your own FQDN\n\u0026ldquo;Mapping type\u0026rdquo;: Expression \u0026ldquo;Expression\u0026rdquo;: \u0026ldquo;Join(\u0026rdquo;@\u0026quot;, NormalizeDiacritics(StripSpaces([displayName])), \u0026ldquo;GROUPS_DOMAIN\u0026rdquo;)\u0026quot; \u0026ldquo;Target attribute\u0026rdquo;: email Configure Entra ID group mappings Towards the bottom, click \u0026ldquo;Add New Mapping\u0026rdquo;\nUse the following options to indicate which OU to provision user accounts in\nMapping type: Constant Constant Value: /Users Target attribute: OrgUnitPath Configure provisioning OU Click OK\nClick Save and confirm\nAssigning user accounts # Note Since I\u0026rsquo;m using the free Azure Entra ID, I cannot sync groups\nOn the Microsoft Entra ID provisioning app, go to Manage -\u0026gt; Users and groups\nAdd one or more users to provision\nClick Select\nClick Assign\nYour result, should look similar to this Provisioning example Enable provisioning # In the Microsoft Entra ID provisioning app, go to Manage -\u0026gt; Provisioning\nClick the \u0026ldquo;Edit Provisioning\u0026rdquo; pencil at the top\nUnder Settings, validate scope is set to \u0026ldquo;Sync only assigned users and groups\u0026rdquo;\nUnder Settings, enable \u0026ldquo;Prevent accidental deletion\u0026rdquo; and enter a threshold of how many accounts the application can delete/disable users automatically\nSet Provisioning Status to \u0026ldquo;On\u0026rdquo; Turning on user provisioning Click Save\nGive the process some time (5-10 minutes) to execute\nOn the provisioning overview page, you should see the status of the provisioning job Entra ID provisioning status In Google Admin, go to Reporting -\u0026gt; Audit and investigation -\u0026gt; Admin log events. You should be able to find a log entry indicating when the account was created in Cloud Identity. Google Admin provisioning status Wrap up # In this article, I created a new relationship with Google Cloud Identity. I used a Microsoft Entra ID application to provision accounts and groups to Google Cloud Identity from Entra ID. In another article, I will provide the steps for configuring SAML/Single sign-on authentication.\n","date":"2024-09","externalUrl":null,"permalink":"/posts/how-to-federate-google-cloud-entra-id-part1/","section":"Posts","summary":"I wanted to use a single account to log into my Azure and Google Cloud environments and automatically provision “source of truth” accounts from Entra ID to Google Cloud Identity. This article will explain how I configured account provisioning of the identity federation solution between Microsoft Entra ID and Google Cloud Identity.\n","title":"Google Cloud federation with Microsoft Entra ID - Part 1","type":"posts"},{"content":"I needed a means of spinning up virtual machines to try out solutions such as Kubernetes or GitLab runners, etc, on a long-term basis. I did not want to incur the cost of running operating systems on Cloud Infrastructure. ESXi was definitely not happening, as Broadcom had muddied the waters at the time. At first, I tried Proxmox, and then I tried Suse Harvester. I contemplated XCP-ng. After weighing what I needed, I settled back to Proxmox VE.\nRequirements # Here are the requirements to follow along\nAt least three compute nodes Each with an Intel or AMD CPU Each with sufficient memory to run your virtual machines Each with a primary and data disks Each with network interfaces capable of 802.1Q VLANs Network switches capable of 802.1Q VLANs A thumb drive or PXE server (I highly recommend netboot.xyz) for the installer An ACME PKI infrastructure is optional Initial boot # If you\u0026rsquo;re using a USB drive for the installation, head over to Proxmox VE\u0026rsquo;s download page. Write the installer to the USB drive and insert it into a USB slot on the computer/server.\nThis part would vary depending on the hardware being used. Here, I am using Supermicro servers. Power on your nodes, boot off the USB drive or use PXE.\nAgree to the EULA\nCreate a ZFS Raid1 mirrored volume for the operating system and Proxmox\nZFS Boot disk configuration Enter your country, time zone, and keyboard layout\nFor the root account, enter a password and email address (which can be fake for now)\nSelect the management network interface and configure the hostname, IP address, gateway, and DNS server\nReview the summary and click Install. Give the process a few minutes to complete the installation and reboot\nPackage management # Proxmox is a mixture of Debian and Proxmox-specific software (targeted to enterprises). However, I do not need the enterprise subscription bits in this situation, so I will modify how Proxmox gets its packages and updates. Perform the following on all nodes.\nDisable the enterprise subscriptions sed -i\u0026#39;.bak\u0026#39; -e \u0026#39;1 s/^/# /\u0026#39; /etc/apt/sources.list.d/pve-enterprise.list sed -i\u0026#39;.bak\u0026#39; -e \u0026#39;1 s/^/# /\u0026#39; /etc/apt/sources.list.d/ceph.list Add \u0026ldquo;non-free-firmware\u0026rdquo; software sources sed -i\u0026#39;.bak\u0026#39; -e \u0026#39;/.debian.org/ s/$/ non-free-firmware/\u0026#39; /etc/apt/sources.list Add the \u0026ldquo;no-subscription\u0026rdquo; software repositories tee --append /etc/apt/sources.list \u0026lt;\u0026lt; EOF \u0026gt; /dev/null # Proxmox VE pve-no-subscription repository provided by proxmox.com, # NOT recommended for production use deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription EOF tee /etc/apt/sources.list.d/ceph.list \u0026lt;\u0026lt; EOF \u0026gt; /dev/null # Ceph Reef no-subscription repository deb http://download.proxmox.com/debian/ceph-reef bookworm no-subscription EOF Install updates, CPU-specific microcode, and then reboot apt-get update apt-get dist-upgrade -y case $(lscpu | awk \u0026#39;/^Vendor ID:/{print $3}\u0026#39;) in \\ \u0026#34;AuthenticAMD\u0026#34;) apt-get install amd64-microcode -y ;; \\ \u0026#34;GenuineIntel\u0026#34;) apt-get install intel-microcode -y ;; \\ esac reboot Networking # I choose not to use Proxmox\u0026rsquo;s SDN feature for networking. Instead, I will use VLANs to leverage my physical switching/routing infrastructure.\nOn each node, back up the network configuration file /etc/network/interfaces cp /etc/network/interfaces /etc/network/interfaces.bak I will write a custom network configuration file to move the management IP address to a dedicated network interface and form a bond with the 10Gb interfaces. I will also create two sub-interfaces for virtual machine migration and Ceph storage replication on the bond.\nLAST_OCTET=`echo $(hostname -i) | cut -d . -f 4` tee /etc/network/interfaces \u0026lt;\u0026lt; EOF \u0026gt; /dev/null auto lo iface lo inet loopback auto eno1 iface eno1 inet static address 192.168.100.${LAST_OCTET}/25 gateway 192.168.100.1 auto eno2 iface eno2 inet manual auto enp67s0f0 iface enp67s0f0 inet manual auto enp67s0f1 iface enp67s0f1 inet manual auto bond0 iface bond0 inet manual bond-slaves enp67s0f0 enp67s0f1 bond-miimon 100 bond-mode 802.3ad bond-xmit-hash-policy layer2+3 auto bond0.102 iface bond0.102 inet static address 192.168.102.${LAST_OCTET}/26 auto bond0.104 iface bond0.104 inet static address 192.168.104.${LAST_OCTET}/26 auto vmbr0 iface vmbr0 inet manual bridge-ports bond0 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 2-4094 source /etc/network/interfaces.d/* EOF Tell the operating system to start using the updated network configuration ifreload --all Time synchronization # On each node, back up the chrony configuration file cp /etc/chrony/chrony.conf /etc/chrony/chrony.conf.bak Then, write out a new configuration file that includes the primary NTP source on an internal IP address and us.pool.ntp.org as the backup tee /etc/chrony/chrony.conf \u0026lt;\u0026lt; EOF \u0026gt; /dev/null server 172.16.2.1 prefer iburst minpoll 2 maxpoll 2 xleave pool us.pool.ntp.org iburst driftfile /var/lib/chrony/chrony.drift makestep 0.1 3 rtcsync keyfile /etc/chrony/chrony.keys logdir /var/log/chrony leapsectz right/UTC EOF Restart the chrony service so the configuration change takes effect right away systemctl restart chronyd ACME Certificates # Add the root certificate from the certificate authority server to each node and update the truststore curl -k https://ca.lab.howto.engineer:443/roots.pem -o /usr/local/share/ca-certificates/root_ca.crt update-ca-certificates On each node, register with the certificate authority server. Substitute \u0026lt;email@tld\u0026gt; with an appropriate email address pvenode acme account register default \u0026lt;email@tld\u0026gt; \\ --directory https://ca.lab.howto.engineer/acme/acme/directory Now set the domain for the acme http-01 challenge pvenode config set --acme domains=$HOSTNAME.lab.howto.engineer Request a certificate for each node pvenode acme cert order Note I am unsure why this happens, but logging into nodes two and three may not work now. Re-running the previous command on the affected nodes rectifies the issue.\nCluster formation # On the first node, create a cluster pvecm create PROXMOX-CLUSTER On the remaining nodes, execute the following. Enter the root password when prompted and accept adding the SSH key fingerprint pvecm add compute-node01.lab.howto.engineer Review the status of the cluster pvecm status Back on the first node, configure cluster-wide settings sed -i \u0026#34;/console:/d\u0026#34; /etc/pve/datacenter.cfg sed -i \u0026#34;/crs:/d\u0026#34; /etc/pve/datacenter.cfg sed -i \u0026#34;/ha:/d\u0026#34; /etc/pve/datacenter.cfg sed -i \u0026#34;/migration:/d\u0026#34; /etc/pve/datacenter.cfg tee --append /etc/pve/datacenter.cfg \u0026lt;\u0026lt; EOF \u0026gt; /dev/null console: html5 crs: ha=static,ha-rebalance-on-start=1 ha: shutdown_policy=migrate migration: insecure,network=192.168.102.0/26 EOF Then create an HA (High Availability) group, giving each node an equal weight ha-manager groupadd ha-global-group \\ --nodes \u0026#34;compute-node01:100,compute-node02:100,compute-node03:100\u0026#34; NFS Backend # I use an NFS share to store ISO and cloud-init files. The next step is to add this share to the cluster.\nOn the first node, validate that the NFS server exports are available to the cluster nodes. The output should list the IP addresses for each node associated with the NFS share.\npvesm nfsscan \u0026lt;NFS-SERVER-IP\u0026gt; | grep -E \u0026#39;192\\.168\\.100\\.2[1-3]\u0026#39; From the first node, run the following. Substitute the values of \u0026lt;NFS-SERVER-IP\u0026gt; and \u0026lt;NFS-SHARE-PATH\u0026gt; for the NFS server and share path details tee /etc/pve/storage.cfg \u0026lt;\u0026lt; EOF \u0026gt; /dev/null nfs: artifacts path /mnt/pve/artifacts server \u0026lt;NFS-SERVER-IP\u0026gt; export \u0026lt;NFS-SHARE-PATH\u0026gt; options vers=4,soft content iso,snippets EOF Ceph storage # On each node, install the required Ceph package. Select \u0026ldquo;y\u0026rdquo; when prompted pveceph install --repository no-subscription --version reef On the first node, initialize Ceph\u0026rsquo;s configuration pveceph init --network 192.168.104.0/26 On each node, create a Ceph monitor and manager pveceph mon create \u0026amp;\u0026amp; pveceph mgr create On each node, validate which available disks will be used for Ceph object storage lsblk Prepare the disks on all nodes by wiping them. This will destroy any data on the disks ceph-volume lvm zap /dev/nvme0n1 --destroy ceph-volume lvm zap /dev/nvme1n1 --destroy Designate the disks to Ceph object storage pveceph osd create /dev/nvme0n1 pveceph osd create /dev/nvme1n1 On the first node, create a Ceph storage pool and assign the available disks to it pveceph pool create \u0026lt;pool-name\u0026gt; --add_storages Wrap up # With those steps out of the way, point your web browser to https://:8006 and log in with the root credentials. This setup is excellent for creating virtual machines for further learning and development without worrying about the monthly cost of similar infrastructure in someone else\u0026rsquo;s data center.\n","date":"2024-08","externalUrl":null,"permalink":"/posts/how-to-setup-a-proxmox-cluster/","section":"Posts","summary":"I needed a means of spinning up virtual machines to try out solutions such as Kubernetes or GitLab runners, etc, on a long-term basis. I did not want to incur the cost of running operating systems on Cloud Infrastructure. ESXi was definitely not happening, as Broadcom had muddied the waters at the time. At first, I tried Proxmox, and then I tried Suse Harvester. I contemplated XCP-ng. After weighing what I needed, I settled back to Proxmox VE.\n","title":"How to Setup a Proxmox Cluster","type":"posts"},{"content":"","date":"2024-08","externalUrl":null,"permalink":"/tags/proxmox/","section":"Tags","summary":"","title":"Proxmox","type":"tags"},{"content":"","date":"2024-07","externalUrl":null,"permalink":"/tags/consul/","section":"Tags","summary":"","title":"Consul","type":"tags"},{"content":"","date":"2024-07","externalUrl":null,"permalink":"/tags/hashicorp/","section":"Tags","summary":"","title":"Hashicorp","type":"tags"},{"content":"So what exactly is Hashicorp Consul? Here is what the Hashicorp has to say:\nHashiCorp Consul is a service networking solution that enables teams to manage secure network connectivity between services and across on-prem and multi-cloud environments and runtimes. Consul offers service discovery, service mesh, traffic management, and automated updates to network infrastructure devices.\nFor the time being, I am targeting Consul\u0026rsquo;s service discovery features. In this article, I will show you how I went about this.\nSo, what exactly is service discovery? According to Wikipedia, it is the process of automatically detecting devices and services on a network. Some of the benefits of service discovery include:\nDynamic IP address and port discovery Load balances requests to healthy service instances Automated service registration and de-registration Consul\u0026rsquo;s service discovery is used alongside other solutions, such as Kubernetes or Nomad, to discover when and where said microservices run dynamically.\nTake the following diagram as a simple example.\nThe application registers itself with Consul with a name called \u0026ldquo;web\u0026rdquo;. Consul will create an internal DNS entry called web.service.consul When a client requests web.service.consul, the load balancer will perform a DNS lookup against Consul. If configured properly, Consul will return all instances of web that are online and healthy The load balancer then forwards the request to the available instances of web Service Discovery example In this diagram, one of the web instances is down for some reason. In this case, Consul will mark it as not available, and DNS queries against web.service.consul will not include the offline node in their response.\nInstallation # Since I have already installed Hashicorp\u0026rsquo;s Vault, I did not have to configure Ubuntu\u0026rsquo;s APT\u0026rsquo;s sources again.\nInstall Consul on all nodes sudo apt update \u0026amp;\u0026amp; \\ sudo apt install --yes consul Validate the installation. It should output the installed version of Consul consul version Now, I want Consul communications to be encrypted at the application level. To do so, an encryption key is required. Fortunately, Consul can generate one for you.\nFrom any server, capture a consul encryption key and store it in a variable CONSUL_ENCRYPT=$(consul kengen) Now, let\u0026rsquo;s define a few configuration files on each server # Node 1 sudo tee /etc/consul.d/consul.hcl \u0026lt;\u0026lt; EOF \u0026gt; /dev/null datacenter = \u0026#34;homelab\u0026#34; data_dir = \u0026#34;/opt/consul/data\u0026#34; encrypt = \u0026#34;${CONSUL_ENCRYPT}\u0026#34; retry_join = [ \u0026#34;192.168.100.11\u0026#34;, \u0026#34;192.168.100.12\u0026#34; ] EOF sudo tee /etc/consul.d/server.hcl \u0026lt;\u0026lt; EOF \u0026gt; /dev/null server = true bind_addr = \u0026#34;192.168.100.10\u0026#34; client_addr = \u0026#34;127.0.0.1 192.168.100.10\u0026#34; ui_config { enabled = true } log_level = \u0026#34;INFO\u0026#34; EOF # Node 2 sudo tee /etc/consul.d/consul.hcl \u0026lt;\u0026lt; EOF \u0026gt; /dev/null datacenter = \u0026#34;homelab\u0026#34; data_dir = \u0026#34;/opt/consul/data\u0026#34; encrypt = \u0026#34;${CONSUL_ENCRYPT}\u0026#34; retry_join = [ \u0026#34;192.168.100.10\u0026#34;, \u0026#34;192.168.100.12\u0026#34; ] EOF sudo tee /etc/consul.d/server.hcl \u0026lt;\u0026lt; EOF \u0026gt; /dev/null server = true bind_addr = \u0026#34;192.168.100.11\u0026#34; client_addr = \u0026#34;127.0.0.1 192.168.100.11\u0026#34; ui_config { enabled = true } log_level = \u0026#34;INFO\u0026#34; EOF # Node 2 sudo tee /etc/consul.d/consul.hcl \u0026lt;\u0026lt; EOF \u0026gt; /dev/null datacenter = \u0026#34;homelab\u0026#34; data_dir = \u0026#34;/opt/consul/data\u0026#34; encrypt = \u0026#34;${CONSUL_ENCRYPT}\u0026#34; retry_join = [ \u0026#34;192.168.100.10\u0026#34;, \u0026#34;192.168.100.11\u0026#34; ] EOF sudo tee /etc/consul.d/server.hcl \u0026lt;\u0026lt; EOF \u0026gt; /dev/null server = true bind_addr = \u0026#34;192.168.100.12\u0026#34; client_addr = \u0026#34;127.0.0.1 192.168.100.12\u0026#34; ui_config { enabled = true } log_level = \u0026#34;INFO\u0026#34; EOF On all servers, validate its configuration sudo consul validate /etc/consul.d/*.hcl On all servers, start the consul service sudo systemctl enable consul; sudo systemctl start consul; sudo systemctl status consul; If you did not receive any errors, point your browser to http://[node-ip]:8500 Consul Homepage If you followed my tutorial to install Hashicorp Vault, part of the configuration included a section where Vault registers itself with Consul. If you look at /etc/vault.d/vault.hcl, you will find the following:\nservice_registration \u0026#34;consul\u0026#34; { address = \u0026#34;http://127.0.0.1:8500\u0026#34; } That configuration is registering Vault in Consul.\nOn the left side, click on Services, and you should see something similar to this:\nConsul Services If you click into the Vault service, you will find several things that Consul is aware of:\nThe Vault instances that are reporting to Consul The status of Vault, i.e., Initialized state or whether a node is the active or stand-by nodes Health check statuses By default, Consul is the authoritative DNS service for the .consul domain. Consul\u0026rsquo;s DNS listens on a non-standard port of 8600 and responds to queries.\nTake, for example:\ndig @192.168.100.10 -p 8600 consul.service.consul ; \u0026lt;\u0026lt;\u0026gt;\u0026gt; DiG 9.10.6 \u0026lt;\u0026lt;\u0026gt;\u0026gt; @192.168.100.10 -p 8600 consul.service.consul ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -\u0026gt;\u0026gt;HEADER\u0026lt;\u0026lt;- opcode: QUERY, status: NOERROR, id: 29631 ;; flags: qr aa rd; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;consul.service.consul.\tIN\tA ;; ANSWER SECTION: consul.service.consul.\t0\tIN\tA\t192.168.100.11 consul.service.consul.\t0\tIN\tA\t192.168.100.12 consul.service.consul.\t0\tIN\tA\t192.168.100.10 ;; Query time: 33 msec ;; SERVER: 192.168.100.10#8600(192.168.100.10) ;; WHEN: Fri Nov 29 13:26:21 EST 2024 ;; MSG SIZE rcvd: 98 Notice in the answer section that Consul responded with all instances of itself.\nDepending on your DNS infrastructure, you can also have your internal DNS servers forward queries to Consul. So instead of querying *.consul, you can query against your FQDN.\ndig consul.lab.howto.engineer +short consul.service.consul. 192.168.100.12 192.168.100.10 192.168.100.11 Wrap up # In this article, I briefly introduced Consul, went through a simple installation, and touched on what its out-of-the-box service discovery could look like. In a future article on Hashicorp Nomad, I will explore how I leverage Consul\u0026rsquo;s service discovery with Nomad\u0026rsquo;s container orchestration.\n","date":"2024-07","externalUrl":null,"permalink":"/posts/how-to-setup-hashicorp-consul/","section":"Posts","summary":"So what exactly is Hashicorp Consul? Here is what the Hashicorp has to say:\nHashiCorp Consul is a service networking solution that enables teams to manage secure network connectivity between services and across on-prem and multi-cloud environments and runtimes. Consul offers service discovery, service mesh, traffic management, and automated updates to network infrastructure devices.\nFor the time being, I am targeting Consul’s service discovery features. In this article, I will show you how I went about this.\n","title":"How to Setup Hashicorp Consul","type":"posts"},{"content":"","date":"2024-07","externalUrl":null,"permalink":"/tags/service-discovery/","section":"Tags","summary":"","title":"Service-Discovery","type":"tags"},{"content":"We have all been there. That newly installed application required confidential material to function. Where should that material be securely stored? Or, you just took over ownership of a system where the database credentials are stored in plain text! We all know (or should know) that protecting secrets is important. Just about anyone, intentional or not, could be a threat actor. Our trust and integrity depend on securing our secrets.\nTo that end, for my immediate needs, I wanted a way to store confidential infrastructure material for:\nApplication API token long term storage CI (continuous integration) and workflow-type scenarios where API tokens are required Initially, I looked at Bitwarden Secrets Manager as a hosted solution. But I read that for their free tier, there was an undocumented API rate limit, that I did not want to be bothered with. Maybe in the future.\nI then turned to Hashicorp\u0026rsquo;s Vault to fulfill my needs. I can run it on my hardware, and it costs me nothing.\nVault is described as:\nHashiCorp Vault is an identity-based secrets and encryption management system. It provides encryption services that are gated by authentication and authorization methods to ensure secure, auditable and restricted access to secrets. Here is what I did to get it installed.\nTo complete this how-to, I used three Raspberry Pi running Ubuntu 24.04. A PKI solution is also required to generate certificates. I use step-ca, so adjust to your circumstance.\nCertificates # On the certificate server, create folders for each server and add the root certificate to them sudo -i mkdir -p ~/certs/prod-core-services0{1,2,3} step ca root ~/certs/prod-core-services01/root_ca.crt \\ --fingerprint $(step certificate fingerprint /etc/step-ca/certs/root_ca.crt) \\ --ca-url https://ca.lab.howto.engineer cp -v ~/certs/prod-core-services01/root_ca.crt certs/prod-core-services02/ cp -v ~/certs/prod-core-services01/root_ca.crt certs/prod-core-services03/ Create certificates with appropriate SAN entries for each server and with 30 days of lifetime before they expire step ca certificate prod-core-services01.lab.howto.engineer \\ ~/certs/prod-core-services01/vault.crt \\ ~/certs/prod-core-services01/vault.key \\ --ca-url https://ca.lab.howto.engineer \\ --root /etc/step-ca/certs/root_ca.crt \\ --not-after=\u0026#34;720h\u0026#34; \\ --san=\u0026#34;prod-core-services01\u0026#34; \\ --san=\u0026#34;192.168.100.10\u0026#34; step ca certificate prod-core-services02.lab.howto.engineer \\ ~/certs/prod-core-services02/vault.crt \\ ~/certs/prod-core-services02/vault.key \\ --ca-url https://ca.lab.howto.engineer \\ --root /etc/step-ca/certs/root_ca.crt \\ --not-after=\u0026#34;720h\u0026#34; \\ --san=\u0026#34;prod-core-services02\u0026#34; \\ --san=\u0026#34;192.168.100.11\u0026#34; step ca certificate prod-core-services03.lab.howto.engineer \\ ~/certs/prod-core-services03/vault.crt \\ ~/certs/prod-core-services03/vault.key \\ --ca-url https://ca.lab.howto.engineer \\ --root /etc/step-ca/certs/root_ca.crt \\ --not-after=\u0026#34;720h\u0026#34; \\ --san=\u0026#34;prod-core-services03\u0026#34; \\ --san=\u0026#34;192.168.100.12\u0026#34; Move the certs directory to a location where it can be copied to destination servers\nFrom another terminal, copy the node specific folder to each destination server\nscp -r ca:~/certs/prod-core-services01 prod-core-services01: scp -r ca:~/certs/prod-core-services02 prod-core-services02: scp -r ca:~/certs/prod-core-services03 prod-core-services03: Remove the certificates from the certificate server find certs -type f -exec shred -u {} \\; rm -rf certs Preparation # On the Vault servers, create a directory for the certificate files sudo mkdir -p /etc/step/certs/vault Copy the certificates to the directory created in the prior step # On prod-core-services01 sudo mv -v ~/prod-core-services01/* /etc/step/certs/vault/ rm -rf ~/prod-core-services01 # On prod-core-services02 sudo mv -v ~/prod-core-services02/* /etc/step/certs/vault/ rm -rf ~/prod-core-services02 # On prod-core-services03 sudo mv -v ~/prod-core-services03/* /etc/step/certs/vault/ rm -rf ~/prod-core-services03 Modify the ownership/permissions on the certificate files sudo chown root:root /etc/step/certs/vault/vault.crt \\ /etc/step/certs/vault/root_ca.crt sudo chown root:vault /etc/step/certs/vault/vault.key sudo chmod 0644 /etc/step/certs/vault/vault.crt \\ /etc/step/certs/vault/root_ca.crt sudo chmod 0640 /etc/step/certs/vault/vault.key If not completed prior, I needed to configure Ubuntu\u0026rsquo;s APT for Hashicorp\u0026rsquo;s software sources on all nodes. sudo apt update \u0026amp;\u0026amp; sudo apt install --yes wget gpg coreutils echo \u0026#34;deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main\u0026#34; | sudo tee /etc/apt/sources.list.d/hashicorp.list wget --quiet -O - https://apt.releases.hashicorp.com/gpg | \\ sudo gpg --dearmor --output \\ /usr/share/keyrings/hashicorp-archive-keyring.gpg Now let me install vault on all nodes. sudo apt update \u0026amp;\u0026amp; sudo apt install --yes vault Grant the vault user ownership of its working directory on all nodes. sudo chown -R vault:vault /opt/vault/ I now have everything required to configure the vault. Let\u0026rsquo;s proceed to the next section.\nConfiguring Hashicorp Vault # Backup the default Vault configuration file on all servers. sudo mv /etc/vault.d/vault.hcl /etc/vault.d/vault.hcl.bak Now, I am going to create a unique configuration file per server. # Node 1 sudo tee /etc/vault.d/vault.hcl \u0026lt;\u0026lt; EOF \u0026gt; /dev/null ui = true cluster_addr = \u0026#34;https://prod-core-services01:8201\u0026#34; api_addr = \u0026#34;https://prod-core-services01:8200\u0026#34; disable_mlock = true storage \u0026#34;raft\u0026#34; { path = \u0026#34;/opt/vault/data\u0026#34; retry_join { leader_tls_servername = \u0026#34;prod-core-services02\u0026#34; leader_api_addr = \u0026#34;https://prod-core-services02:8200\u0026#34; leader_ca_cert_file = \u0026#34;/etc/step/certs/vault/root_ca.crt\u0026#34; leader_client_cert_file = \u0026#34;/etc/step/certs/vault/vault.crt\u0026#34; leader_client_key_file = \u0026#34;/etc/step/certs/vault/vault.key\u0026#34; } retry_join { leader_tls_servername = \u0026#34;prod-core-services03\u0026#34; leader_api_addr = \u0026#34;https://prod-core-services03:8200\u0026#34; leader_ca_cert_file = \u0026#34;/etc/step/certs/vault/root_ca.crt\u0026#34; leader_client_cert_file = \u0026#34;/etc/step/certs/vault/vault.crt\u0026#34; leader_client_key_file = \u0026#34;/etc/step/certs/vault/vault.key\u0026#34; } } listener \u0026#34;tcp\u0026#34; { address = \u0026#34;0.0.0.0:8200\u0026#34; tls_cert_file = \u0026#34;/etc/step/certs/vault/vault.crt\u0026#34; tls_key_file = \u0026#34;/etc/step/certs/vault/vault.key\u0026#34; tls_client_ca_file = \u0026#34;/etc/step/certs/vault/root_ca.crt\u0026#34; } service_registration \u0026#34;consul\u0026#34; { address = \u0026#34;http://127.0.0.1:8500\u0026#34; } EOF # Node 2 sudo tee /etc/vault.d/vault.hcl \u0026lt;\u0026lt; EOF \u0026gt; /dev/null ui = true cluster_addr = \u0026#34;https://prod-core-services02:8201\u0026#34; api_addr = \u0026#34;https://prod-core-services02:8200\u0026#34; disable_mlock = true storage \u0026#34;raft\u0026#34; { path = \u0026#34;/opt/vault/data\u0026#34; retry_join { leader_tls_servername = \u0026#34;prod-core-services01\u0026#34; leader_api_addr = \u0026#34;https://prod-core-services01:8200\u0026#34; leader_ca_cert_file = \u0026#34;/etc/step/certs/vault/root_ca.crt\u0026#34; leader_client_cert_file = \u0026#34;/etc/step/certs/vault/vault.crt\u0026#34; leader_client_key_file = \u0026#34;/etc/step/certs/vault/vault.key\u0026#34; } retry_join { leader_tls_servername = \u0026#34;prod-core-services03\u0026#34; leader_api_addr = \u0026#34;https://prod-core-services03:8200\u0026#34; leader_ca_cert_file = \u0026#34;/etc/step/certs/vault/root_ca.crt\u0026#34; leader_client_cert_file = \u0026#34;/etc/step/certs/vault/vault.crt\u0026#34; leader_client_key_file = \u0026#34;/etc/step/certs/vault/vault.key\u0026#34; } } listener \u0026#34;tcp\u0026#34; { address = \u0026#34;0.0.0.0:8200\u0026#34; tls_cert_file = \u0026#34;/etc/step/certs/vault/vault.crt\u0026#34; tls_key_file = \u0026#34;/etc/step/certs/vault/vault.key\u0026#34; tls_client_ca_file = \u0026#34;/etc/step/certs/vault/root_ca.crt\u0026#34; } service_registration \u0026#34;consul\u0026#34; { address = \u0026#34;http://127.0.0.1:8500\u0026#34; } EOF # Node 3 sudo tee /etc/vault.d/vault.hcl \u0026lt;\u0026lt; EOF \u0026gt; /dev/null ui = true cluster_addr = \u0026#34;https://prod-core-services03:8201\u0026#34; api_addr = \u0026#34;https://prod-core-services03:8200\u0026#34; disable_mlock = true storage \u0026#34;raft\u0026#34; { path = \u0026#34;/opt/vault/data\u0026#34; retry_join { leader_tls_servername = \u0026#34;prod-core-services01\u0026#34; leader_api_addr = \u0026#34;https://prod-core-services01:8200\u0026#34; leader_ca_cert_file = \u0026#34;/etc/step/certs/vault/root_ca.crt\u0026#34; leader_client_cert_file = \u0026#34;/etc/step/certs/vault/vault.crt\u0026#34; leader_client_key_file = \u0026#34;/etc/step/certs/vault/vault.key\u0026#34; } retry_join { leader_tls_servername = \u0026#34;prod-core-services02\u0026#34; leader_api_addr = \u0026#34;https://prod-core-services02:8200\u0026#34; leader_ca_cert_file = \u0026#34;/etc/step/certs/vault/root_ca.crt\u0026#34; leader_client_cert_file = \u0026#34;/etc/step/certs/vault/vault.crt\u0026#34; leader_client_key_file = \u0026#34;/etc/step/certs/vault/vault.key\u0026#34; } } listener \u0026#34;tcp\u0026#34; { address = \u0026#34;0.0.0.0:8200\u0026#34; tls_cert_file = \u0026#34;/etc/step/certs/vault/vault.crt\u0026#34; tls_key_file = \u0026#34;/etc/step/certs/vault/vault.key\u0026#34; tls_client_ca_file = \u0026#34;/etc/step/certs/vault/root_ca.crt\u0026#34; } service_registration \u0026#34;consul\u0026#34; { address = \u0026#34;http://127.0.0.1:8500\u0026#34; } EOF To work with vault from the terminal, a few variables are required to be set. export VAULT_ADDR=\u0026#34;https://$(hostname):8200\u0026#34; export VAULT_CACERT=\u0026#34;/etc/step/certs/vault/vault.crt\u0026#34; export CA_CERT=$(sudo cat /etc/step/certs/vault/certs/root_ca.crt) export DBUS_SESSION_BUS_ADDRESS=/dev/null Start the Vault service and monitor its startup logs. sudo systemctl enable vault sudo systemctl start vault sudo systemctl status vault Check the vault\u0026rsquo;s status. Note the Sealed and Unseal Progress lines. vault status In its current state, the vault is running but sealed or locked. I will require unlock key(s) (something like a password) to unlock it. I\u0026rsquo;ll initialize the vault to get these unlock key(s). Here I am overridding the default values for recovery-shares and recovery-threshold. For more details, please review the documentation for vault operator init. vault operator init -recovery-shares=1 -recovery-threshold=1 Note the output of the previous command. It will generate a list of unlock key(s) and a root token. Once unsealed, the root token is used to log into vault. Save this information in a secure location.\nNow, to unseal the vault, execute the following and enter one of the Vault unseal key(s). vault operator unseal If required, repeat the prior step with additional unseal keys until the vault is unsealed.\nIf all went well, log into Vault with the root token by pointing your browser to one of the Vault servers (https://:8200/ui/) or by using the terminal.\nvault login Certificate maintenance # As noted earlier when I created the certificates, I opted for 30 days before they need to be renewed. To make renewals easier, let me use the step cli and systemd to automate the process.\nInstall the step cli on all nodes wget https://dl.smallstep.com/gh-release/cli/gh-release-header/v0.27.2/step-cli_0.27.2_arm64.deb sudo dpkg -i step-cli_0.27.2_arm64.deb rm step-cli_0.27.2_arm64.deb Write out a systemd service and timer files for scheduling step # Service sudo tee /etc/systemd/system/cert-renewer@.service \u0026lt;\u0026lt; EOF \u0026gt; /dev/null [Unit] Description=Certificate renewer for %I After=network-online.target Documentation=https://smallstep.com/docs/step-ca/certificate-authority-server-production StartLimitIntervalSec=0 PartOf=cert-renewer.target [Service] Type=oneshot User=root Environment=STEPPATH=/etc/step \\ CERT_LOCATION=/etc/step/certs/%i/%i.crt \\ KEY_LOCATION=/etc/step/certs/%i/%i.key ; ExecCondition checks if the certificate is ready for renewal, ; based on the exit status of the command. ; (In systemd \u0026lt;242, you can use ExecStartPre= here.) ExecCondition=/usr/bin/step certificate needs-renewal ${CERT_LOCATION} ; ExecStart renews the certificate, if ExecStartPre was successful. ExecStart=/usr/bin/step ca renew --ca-url https://ca.lab.howto.engineer --force ${CERT_LOCATION} ${KEY_LOCATION} ; Try to reload or restart the systemd service that relies on this cert-renewer ; If the relying service doesn\u0026#39;t exist, forge ahead. ; (In systemd \u0026lt;229, use `reload-or-try-restart` instead of `try-reload-or-restart`) ExecStartPost=/usr/bin/env sh -c \u0026#34;! systemctl --quiet is-active %i.service || systemctl try-reload-or-restart %i\u0026#34; [Install] WantedBy=multi-user.target EOF # Timer sudo tee /etc/systemd/system/cert-renewer@.timer \u0026lt;\u0026lt; EOF \u0026gt; /dev/null [Unit] Description=Timer for certificate renewal of %I Documentation=https://smallstep.com/docs/step-ca/certificate-authority-server-production PartOf=cert-renewer.target [Timer] Persistent=true ; Run the timer unit every 15 minutes. OnCalendar=*:1/15 ; Always run the timer on time. AccuracySec=1us ; Add jitter to prevent a \u0026#34;thundering herd\u0026#34; of simultaneous certificate renewals. RandomizedDelaySec=5m [Install] WantedBy=timers.target EOF Enable the timer service sudo systemctl enable --now cert-renewer@vault.timer List the active timers systemctl list-timers | grep cert-renewer From here on out, step will check every 15 minutes to query the remaining lifetime of the vault certificate. When the default 66% of lifetime has passed, step will start trying to renew the certificate.\nConclusion # In this article, I showed how to:\nCreate certificates using step-ca Installing Hashicorp Vault Configuring Hashicorp Vault using the generated certificates Providing a way to automate certificate renewals Thanks\n","date":"2024-07","externalUrl":null,"permalink":"/posts/configuring-hashicorp-vault/","section":"Posts","summary":"We have all been there. That newly installed application required confidential material to function. Where should that material be securely stored? Or, you just took over ownership of a system where the database credentials are stored in plain text! We all know (or should know) that protecting secrets is important. Just about anyone, intentional or not, could be a threat actor. Our trust and integrity depend on securing our secrets.\n","title":"Configuring Hashicorp Vault","type":"posts"},{"content":"","date":"2024-07","externalUrl":null,"permalink":"/tags/secrets/","section":"Tags","summary":"","title":"Secrets","type":"tags"},{"content":"","date":"2024-07","externalUrl":null,"permalink":"/categories/security/","section":"Categories","summary":"","title":"Security","type":"categories"},{"content":"","date":"2024-07","externalUrl":null,"permalink":"/tags/vault/","section":"Tags","summary":"","title":"Vault","type":"tags"},{"content":"I needed to host an internal PKI (Private Key Infrastructure) to test a secrets management solution. Microsoft Windows PKI requires a complete Active Directory setup, which is overkill for what I needed. Plus, I wanted something open-source. Smallstep\u0026rsquo;s step-ca is open source and is a well-featured private key solution. This post will explain how I set it up using a Nitrokey HSM on a Raspberry Pi 4.\nRequirements # For this project, I am going to use:\n(1) Raspberry Pi 4 2GB Ubuntu 24.04 LTS Nitrokey HSM At least one USB thumb drive to store data offline The rest of this article assumes that the operating system is already functioning.\nInstall pre-requisite software # Log in and get the latest package list sudo apt update Set the hostname sudo hostnamectl set-hostname ca --static Set the timezone sudo timedatectl set-timezone America/New_York Disable the timesyncd service sudo systemctl disable --now systemd-timesyncd Install required packages sudo apt install --yes \\ chrony \\ gcc \\ hsmwiz \\ golang-go \\ gnutls-bin \\ libpcsclite-dev \\ make \\ opensc \\ pkg-config Add the following configuration files for Chrony time synchronization. Here, I have an NTP server at 172.16.2.1. Adjust to your circumstances. sudo tee /etc/chrony/sources.d/homelab-ntp-server.sources \u0026lt;\u0026lt; EOF \u0026gt; /dev/null server 172.16.2.1 prefer iburst minpoll 2 maxpoll 2 xleave pool us.pool.ntp.org iburst maxsources 2 EOF sudo tee /etc/chrony/conf.d/homelab-ntp.conf \u0026lt;\u0026lt; EOF \u0026gt; /dev/null driftfile /var/lib/chrony/chrony.drift makestep 0.1 3 rtcsync keyfile /etc/chrony/chrony.keys logdir /var/log/chrony leapsectz right/UTC EOF Restart the Chrony service sudo systemctl restart chronyd Enable and start the smart card daemon sudo systemctl enable pcscd sudo systemctl start pcscd Install all updates and reboot sudo apt --yes full-upgrade \u0026amp;\u0026amp; reboot All required software, but step/step-ca is now installed. Now, on to configure the solution.\nHSM Configuration and installation of step-ca/step/step-kms-plugin # Insert the USB thumb drive and prepare it sudo fdisk /dev/sda sudo mkfs.ext4 /dev/sda1 -v sudo mount /dev/sda1 /mnt sudo mkdir /mnt/ca sudo chown $USER:$USER /mnt/ca Create a DKEK share and save to the USB thumb drive sc-hsm-tool --create-dkek-share /mnt/ca/dkek-share-stepca.pbe Insert the NitroKey HSM2 and initialize it --so-pin must be composed of 16 hexadecimal characters\n--pin is recommended to be 6 numeric characters\n# If brand-new sudo sc-hsm-tool --initialize \\ --so-pin \u0026lt;new-so-pin\u0026gt; \\ --pin \u0026lt;new-user-pin\u0026gt; \\ --label \u0026#34;step-ca_hsm\u0026#34; \\ --dkek-shares 1 sudo sc-hsm-tool --import-dkek-share /mnt/ca/dkek-share-stepca.pbe # Reformatting back to new and initialize sudo hsmwiz format --so-pin \u0026lt;current-so-pin\u0026gt; sudo sc-hsm-tool --initialize \\ --so-pin 3537363231383830 \\ --pin 648219 \\ --label \u0026#34;step-ca_hsm\u0026#34; \\ --dkek-shares 1 sudo hsmwiz changepin --old 3537363231383830 --new \u0026lt;new-so-pin\u0026gt; --affect-so-pin sudo hsmwiz changepin --old 648219 --new \u0026lt;new-user-pin\u0026gt; # Optional: Verify PINs sudo hsmwiz verifypin --pin \u0026lt;new-user-pin\u0026gt; sudo hsmwiz verifypin --pin \u0026lt;new-so-pin\u0026gt; --verify-sopin sudo sc-hsm-tool --import-dkek-share /mnt/ca/dkek-share-stepca.pbe Download the latest version of step and step-ca (with CGO support) # Install step-ca curl -fsSLO https://github.com/smallstep/certificates/archive/refs/tags/v0.27.2.tar.gz mkdir step-ca tar -xf v0.27.2.tar.gz -C step-ca/ cd step-ca/certificates-0.27.2/ make bootstrap \u0026amp;\u0026amp; make build GO_ENVS=\u0026#34;CGO_ENABLED=1\u0026#34; sudo cp certificates/bin/step-ca /usr/local/ cd ~ \u0026amp;\u0026amp; rm -rf go step-ca *.tar.gz # Validate step-ca step-ca version # Install step-kms-plugin curl -fsSLO https://github.com/smallstep/step-kms-plugin/releases/download/v0.11.4/step-kms-plugin_0.11.4_linux_arm64.tar.gz tar -xf step-kms-plugin_0.11.4_linux_arm64.tar.gz sudo cp step-kms-plugin_0.11.4/step-kms-plugin /usr/local/bin/ # Install step curl -fsSLO https://dl.smallstep.com/gh-release/cli/gh-release-header/v0.27.1/step_linux_0.27.1_armv7.tar.gz tar -xvf step_linux_0.27.1_armv7.tar.gz sudo mv step_0.27.1/bin/step /usr/local/bin # Validate step step version Sweet! Now, let me create the actual PKI.\nCreating the Private Key Infrastructure # Now, let me collect some information so that I can continue.\n# Get the label for the HSM TOKEN=$(sudo pkcs11-tool --list-token-slots | grep \u0026#39;token label\u0026#39; | cut -d \u0026#39;:\u0026#39; -f 2 | xargs) # Get the PKCS11 module path PKCS_MODULE_PATH=$(p11-kit list-modules | grep \u0026#39;opensc-pkcs11.so\u0026#39; | cut -d \u0026#39;:\u0026#39; -f 2 | xargs) # Assign the card\u0026#39;s user-pin to a variable read -s PKCS_USER_PIN # Now construct the full URI PKCS_URI=\u0026#34;pkcs11:module-path=${PKCS_MODULE_PATH};token=${TOKEN}?pin-value=${PKCS_USER_PIN}\u0026#34; Write out a random private key decryption password to the USB disk. This provisioner password will be required when provisioning certificates. echo $(step crypto rand 32 --format=ascii) \u0026gt; /mnt/ca/decrpyt-pass.txt Decide what ID numerical (I believe these are binary) strings you\u0026rsquo;ll use for the root and intermediate keys. In the next few steps, I use 1000 and 1001 for the root and intermediate keys, respectively.\nLet\u0026rsquo;s generate the root private key\nsudo step kms create \\ --json \\ --kty \u0026#34;EC\u0026#34; \\ --crv \u0026#34;P384\u0026#34; \\ --kms \u0026#34;$PKCS_URI\u0026#34; \u0026#34;pkcs11:id=1000;object=root-ca\u0026#34; Now sign the root certificate using the \u0026ldquo;name\u0026rdquo; value from the JSON output sudo step certificate create \\ --profile root-ca \\ --kms \u0026#34;$PKCS_URI\u0026#34; \\ --key \u0026#34;pkcs11:id=1000;object=root-ca\u0026#34; \\ \u0026#34;HTE Root CA\u0026#34; /mnt/ca/root_ca.crt Create the Intermediate private key sudo step kms create \\ --json \\ --kty \u0026#34;EC\u0026#34; \\ --crv \u0026#34;P384\u0026#34; \\ --kms \u0026#34;$PKCS_URI\u0026#34; \u0026#34;pkcs11:id=1001;object=intermediate-ca\u0026#34; Sign the Intermediate certificate using the root certificate and key stored on the HSM # --ca-key is set to the root key id # --key is set to the name of the root key sudo step certificate create --profile intermediate-ca \\ --kms \u0026#34;$PKCS_URI\u0026#34; \\ --ca-kms \u0026#34;$PKCS_URI\u0026#34; \\ --ca /mnt/ca/root_ca.crt \\ --ca-key \u0026#34;pkcs11:id=1000;object=root-ca\u0026#34; \\ --key \u0026#34;pkcs11:id=1001;object=intermediate-ca\u0026#34; \\ \u0026#34;HTE Intermediate CA\u0026#34; /mnt/ca/intermediate_ca.crt List available keys on the Nitrokey HSM and make a note of the \u0026ldquo;ID\u0026rdquo; fields in the output sudo pkcs15-tool --list-keys Export the private key wrappers for root and intermediate private keys, substituting \u0026ldquo;\u0026lt;reference_id\u0026gt;\u0026rdquo; for each key\u0026rsquo;s ID in the previous step sudo sc-hsm-tool \\ --wrap-key /mnt/ca/root_wrap.bin \\ --key-reference \u0026lt;reference_id\u0026gt; \\ --pin \u0026lt;current-user-pin\u0026gt; sudo sc-hsm-tool \\ --wrap-key /mnt/ca/intermediate_wrap.bin \\ --key-reference \u0026lt;reference_id\u0026gt; \\ --pin \u0026lt;current-user-pin\u0026gt; The public key pairs for the certificate authority have been created. Let me configure step-ca to use these.\nConfiguring step-ca # Initialize step-ca, substituting values for: CA_NAME, SERVER_FQDN SERVER_IP_ADDRESS AN_EMAIL_ADDRESS_OR_UNIQUE_STRING sudo mkdir /etc/step-ca export STEPPATH=\u0026#34;/etc/step-ca\u0026#34; sudo --preserve-env step ca init \\ --name=\u0026#34;\u0026lt;CA_NAME\u0026gt;\u0026#34; \\ --dns=\u0026#34;\u0026lt;SERVER_FQDN\u0026gt;,\u0026lt;SERVER_IP_ADDRESS\u0026gt;\u0026#34; \\ --address=\u0026#34;:443\u0026#34; \\ --provisioner=\u0026#34;\u0026lt;AN_EMAIL_ADDRESS_OR_UNIQUE_STRING\u0026gt;\u0026#34; \\ --deployment-type=\u0026#34;standalone\u0026#34; \\ --remote-management \\ --password-file=\u0026#34;/mnt/ca/decrpyt-pass.txt\u0026#34; Delete the default step-ca certificates and move our custom certificates to /etc/step-ca/certs sudo shred -u /etc/step-ca/secrets/root_ca_key sudo shred -u /etc/step-ca/secrets/intermediate_ca_key sudo cp -v /mnt/ca/{root,intermediate}_ca.crt /etc/step-ca/certs/ Reset the step-ca defaults.json file for the custom root certificate export FINGERPRINT=$(step certificate fingerprint /etc/step-ca/certs/root_ca.crt) sed -i -r \u0026#34;s/^(\\s+\\\u0026#34;fingerprint\\\u0026#34;: ).*/\\1\\\u0026#34;${FINGERPRINT}\\\u0026#34;,/\u0026#34; /etc/step-ca/config/defaults.json Write a new configuration file for step-ca, substituting \u0026lt;INTERMEDIATE_KEY_ID\u0026gt; for the value that was chosen earlier sudo tee /etc/step-ca/config/ca.json \u0026lt;\u0026lt; EOF \u0026gt; /dev/null { \u0026#34;root\u0026#34;: \u0026#34;/etc/step-ca/certs/root_ca.crt\u0026#34;, \u0026#34;federatedRoots\u0026#34;: null, \u0026#34;crt\u0026#34;: \u0026#34;/etc/step-ca/certs/intermediate_ca.crt\u0026#34;, \u0026#34;key\u0026#34;: \u0026#34;pkcs11:id=\u0026lt;INTERMEDIATE_KEY_ID\u0026gt;;object=intermediate-ca\u0026#34;, \u0026#34;kms\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;pkcs11\u0026#34;, \u0026#34;uri\u0026#34;: \u0026#34;pkcs11:module-path=/usr/lib/aarch64-linux-gnu/opensc-pkcs11.so;token=step-ca_hsm (UserPIN)?pin-value=${PKCS_USER_PIN}\u0026#34; }, \u0026#34;address\u0026#34;: \u0026#34;:443\u0026#34;, \u0026#34;insecureAddress\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;dnsNames\u0026#34;: [ \u0026#34;\u0026lt;SERVER_FQDN\u0026gt;\u0026#34;, \u0026#34;\u0026lt;SERVER_IP_ADDRESS\u0026gt;\u0026#34; ], \u0026#34;logger\u0026#34;: { \u0026#34;format\u0026#34;: \u0026#34;text\u0026#34; }, \u0026#34;db\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;badgerv2\u0026#34;, \u0026#34;dataSource\u0026#34;: \u0026#34;/etc/step-ca/db\u0026#34;, \u0026#34;badgerFileLoadingMode\u0026#34;: \u0026#34;\u0026#34; }, \u0026#34;authority\u0026#34;: { \u0026#34;enableAdmin\u0026#34;: true }, \u0026#34;tls\u0026#34;: { \u0026#34;cipherSuites\u0026#34;: [ \u0026#34;TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256\u0026#34;, \u0026#34;TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256\u0026#34; ], \u0026#34;minVersion\u0026#34;: 1.2, \u0026#34;maxVersion\u0026#34;: 1.3, \u0026#34;renegotiation\u0026#34;: false } } EOF Perform a smoke test by starting step-ca manually. If everything is configured correctly, you should not get an error and can validate that it is running via journalctl. When satisfied, shut down the process step-ca /etc/step-ca/config/ca.json Optionally, clone the USB drive to have a backup of all configuration material sudo dd if=/dev/sda of=/dev/sdb bs=1M Unmount the USB drive and lock it away sudo umount /mnt Bringing step-ca into production # All software has been installed, certificates created, step-ca configured and tested. Now let me configure step-ca to run under systemd.\nCreate a service account sudo useradd \\ --user-group --system \\ --home-dir /etc/step-ca --shell /bin/false \\ step Allow the service to use ports less than 1024 sudo setcap CAP_NET_BIND_SERVICE=+eip /usr/local/bin/step-ca Let the service account be the owner of all files/folders in /etc/step-ca sudo chown -R step:step /etc/step-ca Create a systemd unit file for step-ca sudo tee /etc/systemd/system/step-ca.service \u0026lt;\u0026lt; EOF \u0026gt; /dev/null [Unit] Description=step-ca service Documentation=https://smallstep.com/docs/step-ca Documentation=https://smallstep.com/docs/step-ca/certificate-authority-server-production After=network-online.target Wants=network-online.target StartLimitIntervalSec=30 StartLimitBurst=3 ConditionFileNotEmpty=/etc/step-ca/config/ca.json [Service] Type=simple User=step Group=step Environment=STEPPATH=/etc/step-ca WorkingDirectory=/etc/step-ca ExecStart=/usr/local/bin/step-ca config/ca.json ExecReload=/bin/kill --signal HUP $MAINPID Restart=on-failure RestartSec=5 TimeoutStopSec=30 StartLimitInterval=30 StartLimitBurst=3 ; Process capabilities \u0026amp; privileges AmbientCapabilities=CAP_NET_BIND_SERVICE CapabilityBoundingSet=CAP_NET_BIND_SERVICE SecureBits=keep-caps NoNewPrivileges=yes ; Sandboxing ProtectSystem=full ProtectHome=true RestrictNamespaces=true RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 PrivateTmp=true PrivateDevices=true ProtectClock=true ProtectControlGroups=true ProtectKernelTunables=true ProtectKernelLogs=true ProtectKernelModules=true LockPersonality=true RestrictSUIDSGID=true RemoveIPC=true RestrictRealtime=true SystemCallFilter=@system-service SystemCallArchitectures=native MemoryDenyWriteExecute=true ReadWriteDirectories=/etc/step-ca/db [Install] WantedBy=multi-user.target EOF Grant the service account rights to access pcsc and the HSM by adding a polkit rule file sudo tee /etc/polkit-1/rules.d/step-ca.rules \u0026lt;\u0026lt; EOF \u0026gt; /dev/null polkit.addRule(function(action, subject) { if (action.id == \u0026#34;org.debian.pcsc-lite.access_pcsc\u0026#34; \u0026amp;\u0026amp; subject.user == \u0026#34;step\u0026#34;) { return polkit.Result.YES; } }); polkit.addRule(function(action, subject) { if (action.id == \u0026#34;org.debian.pcsc-lite.access_card\u0026#34; \u0026amp;\u0026amp; action.lookup(\u0026#34;reader\u0026#34;) == \u0026#39;Nitrokey Nitrokey HSM (DENK03018290000 ) 00 00\u0026#39; \u0026amp;\u0026amp; subject.user == \u0026#34;step\u0026#34;) { return polkit.Result.YES; } }); EOF Enable and start the service. Validate that no errors are produced sudo systemctl daemon-reload sudo systemctl enable --now step-ca sudo journalctl --follow --unit=step-ca Create a test certificate. You will be prompted for the provisioner password that was generated earlier sudo step ca certificate \u0026#34;localhost\u0026#34; localhost.crt localhost.key --ca-url https://ca.lab.howto.engineer --root /etc/step-ca/certs/root_ca.crt # Output below ✔ Provisioner: REDACTED (JWK) [kid: REDACTED] Please enter the password to decrypt the provisioner key: ✔ CA: https://my.server.fqdn ✔ Certificate: localhost.crt ✔ Private Key: localhost.key Conclusion # In this post, I showed how to prepare and install Smallstep\u0026rsquo;s step-ca certificate authority with a hardware HSM from Nitrokey. All running on a Raspberry Pi 4, providing certificate services to your environment. In a future post, I will dive into step-ca features such as ACME and SSH\n","date":"2024-07","externalUrl":null,"permalink":"/posts/getting-started-with-smallstep/","section":"Posts","summary":"I needed to host an internal PKI (Private Key Infrastructure) to test a secrets management solution. Microsoft Windows PKI requires a complete Active Directory setup, which is overkill for what I needed. Plus, I wanted something open-source. Smallstep’s step-ca is open source and is a well-featured private key solution. This post will explain how I set it up using a Nitrokey HSM on a Raspberry Pi 4.\n","title":"Getting Started With Smallstep","type":"posts"},{"content":"","date":"2024-07","externalUrl":null,"permalink":"/tags/nitrokey/","section":"Tags","summary":"","title":"Nitrokey","type":"tags"},{"content":"","date":"2024-07","externalUrl":null,"permalink":"/tags/pki/","section":"Tags","summary":"","title":"Pki","type":"tags"},{"content":"","date":"2024-07","externalUrl":null,"permalink":"/tags/smallstep/","section":"Tags","summary":"","title":"Smallstep","type":"tags"},{"content":"","date":"2023-11","externalUrl":null,"permalink":"/tags/document-ai/","section":"Tags","summary":"","title":"Document-Ai","type":"tags"},{"content":"Continuing from my last post on Document AI, I am going to show what the bulk import experience is like and a peek at the extracted data.\nSchema additions and bulk upload # First, I’ve added additional fields to the document schema to complete labeling the prior document.\nAdding fields to the document’s schema I have also uploaded a few more documents that need to be imported and labeled.\nList of documents to label Bulk Importing # From the Document AI home screen, go to My Processors -\u0026gt; the processor previously created -\u0026gt; Build\nClick the Import Documents button to get started:\nSection to import documents Choose the storage location of the files in Cloud Storage. In this example, I am choosing Unassigned under Data Split. Then click the checkbox for auto-labeling and select a model that will be used to learn about the documents. Click Import. Once completed, click Manage Dataset.\nImport options Labeling en masse # Click the first document\nBeginning document labeling So far mostly everything was mostly captured correctly by the AI model I used when importing the documents. A few issues though.. The discount_amount field was mislabeled, and item_sub_category has a trailing comma. Now take a moment to think about this.. If it were not for the Generative AI feature set, manually capturing all of these fields, on every imported file, would be very time consuming.\nLabeled document (but with minor errors) Now let me rectify the mistakes. I am going to delete the entry for discount_amount and item_sub_category by hovering over the field and choosing delete.\nProcess to remove the value from a label Very dependent on the document layout, one can choose either the bounding box tool or the text selector tool. In this example I am choosing to capture data using the text selector tool. Select the text and choose the relevant field from the drop down. Do the same for the other fields that need correcting.\nProcess to remove the value from a label Here is what the corrected document looks like. I am satisfied now, so I click confirm on all auto-labeled fields and click Mark as Labeled in the bottom left.\nCompleted document after minor tweaks Once I had a couple documents labeled, I went back to Manage Dataset. Highlighted the Labeled line, click to select all labeled documents. Then assign it to the Training set.\nBreaking up documents into Test/Training datasets I am continuing labeling documents and when I have a few more completed, I’ll move those to the Test set, similar to what I did in the previous step. Now is a good time to review label stats of the completed documents thus far. In the lower right click the View Label Stats button:\nHow to get to document label stats Once there, I have the choice of Model based or Template based training. In this example, I am using the Template based option as it has a lower watermark to be valid. The watermark means that I have to have at least three documents in both the training and test sets that each has all the possible labels defined and labeled. In this example, I do not have enough documents for the discount fields. So I am going to continue labeling the rest of the documents.\nHow to get to document label stats Now that I completed labeling documents, I shuffled around documents between the training and test datasets to provide the following:\nThe training to test ratio should be about roughly 4:1 I have enough documents and labels to meet the requirements for the template based training This is primarily required when training a Document AI model. I will discuss this topic in a future post.\nGreen checks for meeting the watermark with labeling Reviewing document data # With all that completed, now what??? Let’s see what was extracted so far. In a new browser tab/window, point to Cloud Shell IDE.\nIn the terminal window below, execute the following commands to setup a Python environment:\nmkdir doc-ai-temp cd doc-ai-temp/ python3 -m venv .venv source .venv/bin/activate pip3 install google-cloud-documentai prettytable touch main.py cloudshell open main.py Copy the following into main.py:\nfrom google.cloud import documentai_v1beta3 from prettytable import PrettyTable def get_document(): ### # TODO: # Before executing uncomment the following three lines and set the variables for your Document AI processor # project_number = \u0026#34;\u0026#34; # The project number where Document AI was deployed into # location = \u0026#34;\u0026#34; # The location of where Document AI was deployed to: us or eu # processor_id = \u0026#34;\u0026#34; # The processor ID that was used for this demo # Create a client client = documentai_v1beta3.DocumentServiceClient() dataset = \u0026#34;projects/{0}/locations/{1}/processors/{2}/dataset\u0026#34;.format(project_number, location, processor_id) # Initialize a document list request list_request = documentai_v1beta3.ListDocumentsRequest( dataset=dataset, ) # Execute the document list request page_result = client.list_documents(request=list_request) # Loop over the list document responses for response in page_result: document_id = response.document_id # Initialize a document get request document_request = documentai_v1beta3.GetDocumentRequest( dataset=dataset, document_id=document_id, ) # Execute the document get request document_response = client.get_document(request=document_request) # Break the loop. I just want the first document break # Use PrettyTable to generate a nice table table = PrettyTable([\u0026#34;Field Name\u0026#34;, \u0026#34;Confidence\u0026#34;, \u0026#34;Field Value\u0026#34;]) table.align[\u0026#34;Field Value\u0026#34;] = \u0026#34;l\u0026#34; table.sortby = \u0026#34;Field Name\u0026#34; # Loop over document.entities to get the document fields/data and add rows to our table for item in document_response.document.entities: table.add_row([item.type_, item.confidence, item.mention_text]) # Print the table print(table) if __name__ == \u0026#34;__main__\u0026#34;: get_document() In the Google Cloud console, I am grabbing the project_id, location and processor_id from the Document AI processor Overview page under Prediction. Make a note of the values and edit main.py in the TODO section to set these variables:\nHow to get processor information In the terminal window execute: python3 main.py\nYou may get a popup to authorize cloud shell. Click Authorize. You should get output similar to this:\n(.venv) admin_@cloudshell:~/doc-ai-temp$ python main.py +-------------------+------------+-------------------------------------------------+ | Field Name | Confidence | Field Value | +-------------------+------------+-------------------------------------------------+ | currency | 1.0 | $ | | discount_amount | 1.0 | 20 | | discount_total | 1.0 | 35.30 | | freight_total | 1.0 | 4.31 | | grand_total | 1.0 | 145.52 | | invoice_date | 1.0 | Jul 31 2012 | | invoice_id | 1.0 | 33135 | | item_category | 1.0 | Furniture | | item_description | 1.0 | 9-3/4 Diameter Round Wall Clock | | item_product_code | 1.0 | FUR-FU-2877 | | item_quantity | 1.0 | 4 | | item_sub_category | 1.0 | Furnishings | | item_total_price | 1.0 | 176.51 | | item_unit_price | 1.0 | 44.13 | | order_id | 1.0 | CA-2012-BF10975140-41121 | | receiver_address | 1.0 | 28205, Charlotte, North Carolina, United States | | receiver_name | 1.0 | Barbara Fisher | | shipment_mode | 1.0 | Standard Class | | sub_total | 1.0 | 176.51 | +-------------------+------------+-------------------------------------------------+ With the ability to capture data like this from a document, here are some use cases that can occur:\nDocument data can be inserted into databases (BigQuery, MySQL, SQLServer) for fast and targeting querying. Document data can be analyzed like regular data. Document data can be integrated with other data sources like Workday or Salesforce. Document data can be used in conversations via chat-bots and natural language processing search. Document data can be used in multi-lingual use-cases. Document data can be used to detect fraud. Add your why here.. Conclusion # I have shown how to upload and leverage Document AI Generative AI feature set to quickly label documents in bulk. I also touched on label stats while labeling documents. Label statistics will become clearer when training a custom model in situations where a pre-built model is not enough. What I have shown so far, is just the introduction. Document AI has a list of purpose built processors for a wide variety of documents or build a custom one as I did here. For a list of processor pricing, review this page.\nIn a future post, I will use Document AI to develop a workflow process with other solutions to show how the extracted data can be used.\nThanks\n","date":"2023-11","externalUrl":null,"permalink":"/posts/google-document-ai-bulk-import-and-results/","section":"Posts","summary":"Continuing from my last post on Document AI, I am going to show what the bulk import experience is like and a peek at the extracted data.\n","title":"Google Document AI: Bulk Import and Results","type":"posts"},{"content":"","date":"2023-11","externalUrl":null,"permalink":"/tags/machine-learning/","section":"Tags","summary":"","title":"Machine-Learning","type":"tags"},{"content":"Document AI is a Google Cloud solution that imports structured data from unstructured or semi structured documents. The output can then be treated as first class data citizens for analysis with your other data sources to gain deeper insight from your “dark” document data. In this first post on Document AI, I will go over the initial steps to get started.\nRequirements # In this post, I am assuming that you have an account and some basic knowledge of Google Cloud’s console.\nLog into Google Cloud’s console.\nFollow this link to create a project. [Documentation]\nGive the project a name Choose your billing account Follow this link to enable the Document AI API.\nNote: In this screenshot, I have already enabled it. Here is where the API can be enabled or disabled.\nEnabling (or disabling) the Document AI API Follow this link to create a Cloud Storage bucket to store documents. Give it a name. These are naming considerations to be aware of. Click on Create to accept the defaults.\nGo into the storage bucket that was just created and create two folders:\ndataset documents Upload any test files that you may have into the documents folder. Searching for sample documents? Try Kaggle.\nCreating a custom processor # Once completed, navigate to Document AI Workbench and create a new custom extractor. Give the processor a name and an appropriate location. For storage, choose that you will use your own storage and pick the dataset folder that was created earlier. Once satisfied, click Create.\nCreating a custom extractor in Document AI Now I am going to import a sample document.\nImporting a document # Click on “Get Started”\nClicking “Get started” on the Customize card Click on “Upload Sample Document”. Choose to import a file from Cloud Storage, choose the location where the documents reside, highlight one file and click “Import”.\nUploading a sample document Document AI will analyze the document and present a view of it. Once the document is displayed, I will proceed to creating fields and labeling.\nLabeling a document # In this tutorial, I am going to capture the invoice number, the receiver’s name and address, invoice Id and date, shipment method and currency.\nClick the “Create new field” button in the upper left. Make sure to give the field a name, an appropriate data type and for now choose “Optional Once” in the Occurrence drop down and click Create.\nCreating your first label Depending on the document and layout, Document AI’s Generative AI features may auto-label an area (highlighted in light purple) when a field has been created. Continue adding fields you want to capture.\nDisplaying initial document labeling state In the example above, the invoice number was not automatically captured by Generative AI. Here we need to manually draw a bounding box and assign it to the appropriate field. Once labeled, in the lower left click on the “Mark As Labeled” button. Document AI then stores the document along with the data I am extracting from the document.\nManually capturing data on a document Conclusion # I went over how to enable the Document AI service, created a processor from Document AI Workbench and initial importing/labeling of a document. We saw how Generative AI was able to quickly recognize the correct information from the document. This will come into play when we have to train a model with many documents in my next post on Document AI.\nThanks!\n","date":"2023-10","externalUrl":null,"permalink":"/posts/google-document-ai-how-to-get-started/","section":"Posts","summary":"Document AI is a Google Cloud solution that imports structured data from unstructured or semi structured documents. The output can then be treated as first class data citizens for analysis with your other data sources to gain deeper insight from your “dark” document data. In this first post on Document AI, I will go over the initial steps to get started.\n","title":"Google Document AI: How to Get Started","type":"posts"},{"content":" Hello, I\u0026rsquo;m Gerard. # I am a Cloud Engineer with over 15 years of experience architecting, building, and operating infrastructure at scale. I currently work as a Cloud Engineer Consultant at Riverflow Enterprises, LLC, where I help organizations design and implement cloud-native solutions. Previously, I was a Google Cloud Engineer at Google Cloud, a Cloud Operations Manager/Engineer at Total Wine \u0026amp; More, and a Systems Architect/Engineer at the Seminole Tribe of Florida.\nI specialize in the cloud infrastructure and platform engineering space — Kubernetes, cloud networking, CI/CD pipelines, and everything in between. My work spans the full stack of cloud operations: from designing hybrid-cloud networks and managing multi-cloud identity federation, to deploying production-grade Kubernetes clusters and building scalable document processing pipelines on Google Cloud.\nAbout This Blog # This site is where I document what I learn. Each post is a technical guide rooted in real-world experience — the kind of thing I wish I had found when I was working through a problem myself. You will find deep dives on topics like:\nKubernetes — cluster lifecycle, CNI configuration, ingress, service mesh Cloud Platforms — Google Cloud, workload identity, federation, Document AI Homelab Infrastructure — Proxmox, Talos Linux, Cilium, BGP, GitLab runners Security \u0026amp; Identity — OIDC federation, workload identities, certificate management Automation — Infrastructure as Code, Docker, CI/CD pipelines All of the configurations, manifests, and commands on this site have been tested in live environments — either in production or in my homelab.\nApproach # I believe infrastructure should be:\nReproducible — if you cannot automate it, you do not own it Observable — if you cannot measure it, you cannot improve it Minimal — every component should justify its existence Secure by default — shift-left on security, not afterthought Outside of Work # When I am not deep in a terminal session, you can find me capturing moments through my lens, exploring new places, or indulging in local culinary delights. I’m also an avid cyclist, aviation enthusiast, and a firm believer in financial literacy.\nWant to connect or collaborate? Find me on LinkedIn or reach out via the links throughout this site.\n","externalUrl":null,"permalink":"/about/","section":"Gerard Samuel","summary":"","title":"About","type":"page"},{"content":"","externalUrl":null,"permalink":"/authors/","section":"Authors","summary":"","title":"Authors","type":"authors"},{"content":" Company Link Role Dates Location Riverflow Enterprises, LLC Cloud Engineer Consultant Feb 2024 – Present United States Google Cloud Google Cloud Engineer Jun 2022 – Dec 2023 Miami, FL, United States Total Wine and more Cloud Operations Manager/Engineer Sep 2016 – Jun 2022 Boynton Beach, FL \u0026 Bethesda, MD, United States Seminole Tribe of Florida Systems Architect/Engineer Jan 2009 – Aug 2016 Hollywood, FL, United States ","externalUrl":null,"permalink":"/resume/","section":"Gerard Samuel","summary":"","title":"Resume","type":"page"},{"content":"","externalUrl":null,"permalink":"/series/","section":"Series","summary":"","title":"Series","type":"series"}]