Configuring Hashicorp Vault

Table of Contents

We have all been there. That newly installed application required confidential material to function. Where should that material be securely stored? Or, you just took over ownership of a system where the database credentials are stored in plain text! We all know (or should know) that protecting secrets is important. Just about anyone, intentional or not, could be a threat actor. Our trust and integrity depend on securing our secrets.

To that end, for my immediate needs, I wanted a way to store confidential infrastructure material for:

Application API token long term storage
CI (continuous integration) and workflow-type scenarios where API tokens are required

Initially, I looked at Bitwarden Secrets Manager as a hosted solution. But I read that for their free tier, there was an undocumented API rate limit, that I did not want to be bothered with. Maybe in the future.

I then turned to Hashicorp’s Vault to fulfill my needs. I can run it on my hardware, and it costs me nothing.

Vault is described as:

HashiCorp Vault is an identity-based secrets and encryption management system. It provides encryption services that are gated by authentication and authorization methods to ensure secure, auditable and restricted access to secrets.

Here is what I did to get it installed.

To complete this how-to, I used three Raspberry Pi running Ubuntu 24.04. A PKI solution is also required to generate certificates. I use step-ca, so adjust to your circumstance.

Certificates
#

On the certificate server, create folders for each server and add the root certificate to them

sudo -i

mkdir -p ~/certs/prod-core-services0{1,2,3}

step ca root ~/certs/prod-core-services01/root_ca.crt \
--fingerprint $(step certificate fingerprint /etc/step-ca/certs/root_ca.crt) \
--ca-url https://ca.lab.howto.engineer

cp -v ~/certs/prod-core-services01/root_ca.crt certs/prod-core-services02/
cp -v ~/certs/prod-core-services01/root_ca.crt certs/prod-core-services03/

Create certificates with appropriate SAN entries for each server and with 30 days of lifetime before they expire

step ca certificate prod-core-services01.lab.howto.engineer \
  ~/certs/prod-core-services01/vault.crt \
  ~/certs/prod-core-services01/vault.key \
  --ca-url https://ca.lab.howto.engineer \
  --root /etc/step-ca/certs/root_ca.crt \
  --not-after="720h" \
  --san="prod-core-services01" \
  --san="192.168.100.10"

step ca certificate prod-core-services02.lab.howto.engineer \
  ~/certs/prod-core-services02/vault.crt \
  ~/certs/prod-core-services02/vault.key \
  --ca-url https://ca.lab.howto.engineer \
  --root /etc/step-ca/certs/root_ca.crt \
  --not-after="720h" \
  --san="prod-core-services02" \
  --san="192.168.100.11"

step ca certificate prod-core-services03.lab.howto.engineer \
  ~/certs/prod-core-services03/vault.crt \
  ~/certs/prod-core-services03/vault.key \
  --ca-url https://ca.lab.howto.engineer \
  --root /etc/step-ca/certs/root_ca.crt \
  --not-after="720h" \
  --san="prod-core-services03" \
  --san="192.168.100.12"

Move the certs directory to a location where it can be copied to destination servers
From another terminal, copy the node specific folder to each destination server

scp -r ca:~/certs/prod-core-services01 prod-core-services01:
scp -r ca:~/certs/prod-core-services02 prod-core-services02:
scp -r ca:~/certs/prod-core-services03 prod-core-services03:

Remove the certificates from the certificate server

find certs -type f -exec shred -u {} \;
rm -rf certs

Preparation
#

On the Vault servers, create a directory for the certificate files

sudo mkdir -p /etc/step/certs/vault

Copy the certificates to the directory created in the prior step

# On prod-core-services01
sudo mv -v ~/prod-core-services01/* /etc/step/certs/vault/
rm -rf ~/prod-core-services01

# On prod-core-services02
sudo mv -v ~/prod-core-services02/* /etc/step/certs/vault/
rm -rf ~/prod-core-services02

# On prod-core-services03
sudo mv -v ~/prod-core-services03/* /etc/step/certs/vault/
rm -rf ~/prod-core-services03

Modify the ownership/permissions on the certificate files

sudo chown root:root /etc/step/certs/vault/vault.crt \
  /etc/step/certs/vault/root_ca.crt
sudo chown root:vault /etc/step/certs/vault/vault.key
sudo chmod 0644 /etc/step/certs/vault/vault.crt \
  /etc/step/certs/vault/root_ca.crt
sudo chmod 0640 /etc/step/certs/vault/vault.key

If not completed prior, I needed to configure Ubuntu’s APT for Hashicorp’s software sources on all nodes.

sudo apt update && sudo apt install --yes wget gpg coreutils

echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list

wget --quiet -O - https://apt.releases.hashicorp.com/gpg | \
  sudo gpg --dearmor --output \
  /usr/share/keyrings/hashicorp-archive-keyring.gpg

Now let me install vault on all nodes.

sudo apt update && sudo apt install --yes vault

Grant the vault user ownership of its working directory on all nodes.

sudo chown -R vault:vault /opt/vault/

I now have everything required to configure the vault. Let’s proceed to the next section.

Configuring Hashicorp Vault
#

Backup the default Vault configuration file on all servers.

sudo mv /etc/vault.d/vault.hcl /etc/vault.d/vault.hcl.bak

Now, I am going to create a unique configuration file per server.

# Node 1
sudo tee /etc/vault.d/vault.hcl << EOF > /dev/null
ui            = true
cluster_addr  = "https://prod-core-services01:8201"
api_addr      = "https://prod-core-services01:8200"
disable_mlock = true

storage "raft" {
  path    = "/opt/vault/data"

  retry_join {
    leader_tls_servername   = "prod-core-services02"
    leader_api_addr         = "https://prod-core-services02:8200"
    leader_ca_cert_file     = "/etc/step/certs/vault/root_ca.crt"
    leader_client_cert_file = "/etc/step/certs/vault/vault.crt"
    leader_client_key_file  = "/etc/step/certs/vault/vault.key"
  }
  retry_join {
    leader_tls_servername   = "prod-core-services03"
    leader_api_addr         = "https://prod-core-services03:8200"
    leader_ca_cert_file     = "/etc/step/certs/vault/root_ca.crt"
    leader_client_cert_file = "/etc/step/certs/vault/vault.crt"
    leader_client_key_file  = "/etc/step/certs/vault/vault.key"
  }
}

listener "tcp" {
  address            = "0.0.0.0:8200"
  tls_cert_file      = "/etc/step/certs/vault/vault.crt"
  tls_key_file       = "/etc/step/certs/vault/vault.key"
  tls_client_ca_file = "/etc/step/certs/vault/root_ca.crt"
}

service_registration "consul" {
  address      = "http://127.0.0.1:8500"
}
EOF

# Node 2
sudo tee /etc/vault.d/vault.hcl << EOF > /dev/null
ui            = true
cluster_addr  = "https://prod-core-services02:8201"
api_addr      = "https://prod-core-services02:8200"
disable_mlock = true

storage "raft" {
  path    = "/opt/vault/data"

  retry_join {
    leader_tls_servername   = "prod-core-services01"
    leader_api_addr         = "https://prod-core-services01:8200"
    leader_ca_cert_file     = "/etc/step/certs/vault/root_ca.crt"
    leader_client_cert_file = "/etc/step/certs/vault/vault.crt"
    leader_client_key_file  = "/etc/step/certs/vault/vault.key"
  }
  retry_join {
    leader_tls_servername   = "prod-core-services03"
    leader_api_addr         = "https://prod-core-services03:8200"
    leader_ca_cert_file     = "/etc/step/certs/vault/root_ca.crt"
    leader_client_cert_file = "/etc/step/certs/vault/vault.crt"
    leader_client_key_file  = "/etc/step/certs/vault/vault.key"
  }
}

listener "tcp" {
  address            = "0.0.0.0:8200"
  tls_cert_file      = "/etc/step/certs/vault/vault.crt"
  tls_key_file       = "/etc/step/certs/vault/vault.key"
  tls_client_ca_file = "/etc/step/certs/vault/root_ca.crt"
}

service_registration "consul" {
  address      = "http://127.0.0.1:8500"
}
EOF

# Node 3
sudo tee /etc/vault.d/vault.hcl << EOF > /dev/null
ui            = true
cluster_addr  = "https://prod-core-services03:8201"
api_addr      = "https://prod-core-services03:8200"
disable_mlock = true

storage "raft" {
  path    = "/opt/vault/data"

  retry_join {
    leader_tls_servername   = "prod-core-services01"
    leader_api_addr         = "https://prod-core-services01:8200"
    leader_ca_cert_file     = "/etc/step/certs/vault/root_ca.crt"
    leader_client_cert_file = "/etc/step/certs/vault/vault.crt"
    leader_client_key_file  = "/etc/step/certs/vault/vault.key"
  }
  retry_join {
    leader_tls_servername   = "prod-core-services02"
    leader_api_addr         = "https://prod-core-services02:8200"
    leader_ca_cert_file     = "/etc/step/certs/vault/root_ca.crt"
    leader_client_cert_file = "/etc/step/certs/vault/vault.crt"
    leader_client_key_file  = "/etc/step/certs/vault/vault.key"
  }
}

listener "tcp" {
  address            = "0.0.0.0:8200"
  tls_cert_file      = "/etc/step/certs/vault/vault.crt"
  tls_key_file       = "/etc/step/certs/vault/vault.key"
  tls_client_ca_file = "/etc/step/certs/vault/root_ca.crt"
}

service_registration "consul" {
  address      = "http://127.0.0.1:8500"
}
EOF

To work with vault from the terminal, a few variables are required to be set.

export VAULT_ADDR="https://$(hostname):8200"
export VAULT_CACERT="/etc/step/certs/vault/vault.crt"
export CA_CERT=$(sudo cat /etc/step/certs/vault/certs/root_ca.crt)
export DBUS_SESSION_BUS_ADDRESS=/dev/null

Start the Vault service and monitor its startup logs.

sudo systemctl enable vault
sudo systemctl start vault
sudo systemctl status vault

Check the vault’s status. Note the Sealed and Unseal Progress lines.

vault status

In its current state, the vault is running but sealed or locked. I will require unlock key(s) (something like a password) to unlock it. I’ll initialize the vault to get these unlock key(s). Here I am overridding the default values for recovery-shares and recovery-threshold. For more details, please review the documentation for vault operator init.

vault operator init -recovery-shares=1 -recovery-threshold=1

Note the output of the previous command. It will generate a list of unlock key(s) and a root token. Once unsealed, the root token is used to log into vault. Save this information in a secure location.

Now, to unseal the vault, execute the following and enter one of the Vault unseal key(s).

vault operator unseal

If required, repeat the prior step with additional unseal keys until the vault is unsealed.
If all went well, log into Vault with the root token by pointing your browser to one of the Vault servers (https://:8200/ui/) or by using the terminal.

vault login

Certificate maintenance
#

As noted earlier when I created the certificates, I opted for 30 days before they need to be renewed. To make renewals easier, let me use the step cli and systemd to automate the process.

Install the step cli on all nodes

wget https://dl.smallstep.com/gh-release/cli/gh-release-header/v0.27.2/step-cli_0.27.2_arm64.deb
sudo dpkg -i step-cli_0.27.2_arm64.deb
rm step-cli_0.27.2_arm64.deb

Write out a systemd service and timer files for scheduling step

# Service
sudo tee /etc/systemd/system/[email protected] << EOF > /dev/null
[Unit]
Description=Certificate renewer for %I
After=network-online.target
Documentation=https://smallstep.com/docs/step-ca/certificate-authority-server-production
StartLimitIntervalSec=0
PartOf=cert-renewer.target

[Service]
Type=oneshot
User=root

Environment=STEPPATH=/etc/step \
            CERT_LOCATION=/etc/step/certs/%i/%i.crt \
            KEY_LOCATION=/etc/step/certs/%i/%i.key

; ExecCondition checks if the certificate is ready for renewal,
; based on the exit status of the command.
; (In systemd <242, you can use ExecStartPre= here.)
ExecCondition=/usr/bin/step certificate needs-renewal ${CERT_LOCATION}

; ExecStart renews the certificate, if ExecStartPre was successful.
ExecStart=/usr/bin/step ca renew --ca-url https://ca.lab.howto.engineer --force ${CERT_LOCATION} ${KEY_LOCATION}

; Try to reload or restart the systemd service that relies on this cert-renewer
; If the relying service doesn't exist, forge ahead.
; (In systemd <229, use `reload-or-try-restart` instead of `try-reload-or-restart`)
ExecStartPost=/usr/bin/env sh -c "! systemctl --quiet is-active %i.service || systemctl try-reload-or-restart %i"

[Install]
WantedBy=multi-user.target
EOF

# Timer
sudo tee /etc/systemd/system/[email protected] << EOF > /dev/null
[Unit]
Description=Timer for certificate renewal of %I
Documentation=https://smallstep.com/docs/step-ca/certificate-authority-server-production
PartOf=cert-renewer.target

[Timer]
Persistent=true

; Run the timer unit every 15 minutes.
OnCalendar=*:1/15

; Always run the timer on time.
AccuracySec=1us

; Add jitter to prevent a "thundering herd" of simultaneous certificate renewals.
RandomizedDelaySec=5m

[Install]
WantedBy=timers.target
EOF

Enable the timer service

sudo systemctl enable --now [email protected]

List the active timers

systemctl list-timers | grep cert-renewer

From here on out, step will check every 15 minutes to query the remaining lifetime of the vault certificate. When the default 66% of lifetime has passed, step will start trying to renew the certificate.

Conclusion
#

In this article, I showed how to:

Create certificates using step-ca
Installing Hashicorp Vault
Configuring Hashicorp Vault using the generated certificates
Providing a way to automate certificate renewals

Thanks

Certificates#

Preparation#

Configuring Hashicorp Vault#

Certificate maintenance#

Conclusion#

Certificates
#

Preparation
#

Configuring Hashicorp Vault
#

Certificate maintenance
#

Conclusion
#