Configuring Hashicorp Vault

Estimated reading time: 7 minutes

Gerard Samuel Gerard Samuel's profile photo
Image generated by Google Gemini

We have all been there. That newly installed application required confidential material to function. Where should that material be securely stored? Or, you just took over ownership of a system where the database credentials are stored in plain text! We all know (or should know) that protecting secrets is important. Just about anyone, intentional or not, could be a threat actor. Our trust and integrity depend on securing our secrets.

To that end, for my immediate needs, I wanted a way to store confidential infrastructure material for:

  • Application API token long term storage
  • CI (continuous integration) and workflow-type scenarios where API tokens are required

Initially, I looked at Bitwarden Secrets Manager as a hosted solution. But I read that for their free tier, there was an undocumented API rate limit, that I did not want to be bothered with. Maybe in the future.

I then turned to Hashicorp’s Vault to fulfill my needs. I can run it on my hardware, and it costs me nothing.

Vault is described as:

  • HashiCorp Vault is an identity-based secrets and encryption management system. It provides encryption services that are gated by authentication and authorization methods to ensure secure, auditable and restricted access to secrets.

Here is what I did to get it installed.

To complete this how-to, I used three Raspberry Pi running Ubuntu 24.04. A PKI solution is also required to generate certificates. I use step-ca, so adjust to your circumstance.

Certificates

  • On the certificate server, create folders for each server and add the root certificate to them
sudo -i

mkdir -p ~/certs/prod-core-services0{1,2,3}

step ca root ~/certs/prod-core-services01/root_ca.crt \
--fingerprint $(step certificate fingerprint /etc/step-ca/certs/root_ca.crt) \
--ca-url https://ca.lab.howto.engineer

cp -v ~/certs/prod-core-services01/root_ca.crt certs/prod-core-services02/
cp -v ~/certs/prod-core-services01/root_ca.crt certs/prod-core-services03/
  • Create certificates with appropriate SAN entries for each server and with 30 days of lifetime before they expire
step ca certificate prod-core-services01.lab.howto.engineer \
  ~/certs/prod-core-services01/vault.crt \
  ~/certs/prod-core-services01/vault.key \
  --ca-url https://ca.lab.howto.engineer \
  --root /etc/step-ca/certs/root_ca.crt \
  --not-after="720h" \
  --san="prod-core-services01" \
  --san="192.168.100.10"

step ca certificate prod-core-services02.lab.howto.engineer \
  ~/certs/prod-core-services02/vault.crt \
  ~/certs/prod-core-services02/vault.key \
  --ca-url https://ca.lab.howto.engineer \
  --root /etc/step-ca/certs/root_ca.crt \
  --not-after="720h" \
  --san="prod-core-services02" \
  --san="192.168.100.11"

step ca certificate prod-core-services03.lab.howto.engineer \
  ~/certs/prod-core-services03/vault.crt \
  ~/certs/prod-core-services03/vault.key \
  --ca-url https://ca.lab.howto.engineer \
  --root /etc/step-ca/certs/root_ca.crt \
  --not-after="720h" \
  --san="prod-core-services03" \
  --san="192.168.100.12"

  • Move the certs directory to a location where it can be copied to destination servers

  • From another terminal, copy the node specific folder to each destination server

scp -r ca:~/certs/prod-core-services01 prod-core-services01:
scp -r ca:~/certs/prod-core-services02 prod-core-services02:
scp -r ca:~/certs/prod-core-services03 prod-core-services03:
  • Remove the certificates from the certificate server
find certs -type f -exec shred -u {} \;
rm -rf certs

Preparation

  • On the Vault servers, create a directory for the certificate files
sudo mkdir -p /etc/step/certs/vault
  • Copy the certificates to the directory created in the prior step
# On prod-core-services01
sudo mv -v ~/prod-core-services01/* /etc/step/certs/vault/
rm -rf ~/prod-core-services01

# On prod-core-services02
sudo mv -v ~/prod-core-services02/* /etc/step/certs/vault/
rm -rf ~/prod-core-services02

# On prod-core-services03
sudo mv -v ~/prod-core-services03/* /etc/step/certs/vault/
rm -rf ~/prod-core-services03
  • Modify the ownership/permissions on the certificate files
sudo chown root:root /etc/step/certs/vault/vault.crt \
  /etc/step/certs/vault/root_ca.crt
sudo chown root:vault /etc/step/certs/vault/vault.key
sudo chmod 0644 /etc/step/certs/vault/vault.crt \
  /etc/step/certs/vault/root_ca.crt
sudo chmod 0640 /etc/step/certs/vault/vault.key
  • If not completed prior, I needed to configure Ubuntu’s APT for Hashicorp’s software sources on all nodes.
sudo apt update && sudo apt install --yes wget gpg coreutils

echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list

wget --quiet -O - https://apt.releases.hashicorp.com/gpg | \
  sudo gpg --dearmor --output \
  /usr/share/keyrings/hashicorp-archive-keyring.gpg
  • Now let me install vault on all nodes.
sudo apt update && sudo apt install --yes vault
  • Grant the vault user ownership of its working directory on all nodes.
sudo chown -R vault:vault /opt/vault/

I now have everything required to configure the vault. Let’s proceed to the next section.

Configuring Hashicorp Vault

  • Backup the default Vault configuration file on all servers.
sudo mv /etc/vault.d/vault.hcl /etc/vault.d/vault.hcl.bak
  • Now, I am going to create a unique configuration file per server.
# Node 1
sudo tee /etc/vault.d/vault.hcl << EOF > /dev/null
ui            = true
cluster_addr  = "https://prod-core-services01:8201"
api_addr      = "https://prod-core-services01:8200"
disable_mlock = true

storage "raft" {
  path    = "/opt/vault/data"
  
  retry_join {
    leader_tls_servername   = "prod-core-services02"
    leader_api_addr         = "https://prod-core-services02:8200"
    leader_ca_cert_file     = "/etc/step/certs/vault/root_ca.crt"
    leader_client_cert_file = "/etc/step/certs/vault/vault.crt"
    leader_client_key_file  = "/etc/step/certs/vault/vault.key"
  }
  retry_join {
    leader_tls_servername   = "prod-core-services03"
    leader_api_addr         = "https://prod-core-services03:8200"
    leader_ca_cert_file     = "/etc/step/certs/vault/root_ca.crt"
    leader_client_cert_file = "/etc/step/certs/vault/vault.crt"
    leader_client_key_file  = "/etc/step/certs/vault/vault.key"
  }
}

listener "tcp" {
  address            = "0.0.0.0:8200"
  tls_cert_file      = "/etc/step/certs/vault/vault.crt"
  tls_key_file       = "/etc/step/certs/vault/vault.key"
  tls_client_ca_file = "/etc/step/certs/vault/root_ca.crt"
}

service_registration "consul" {
  address      = "http://127.0.0.1:8500"
}
EOF

# Node 2
sudo tee /etc/vault.d/vault.hcl << EOF > /dev/null
ui            = true
cluster_addr  = "https://prod-core-services02:8201"
api_addr      = "https://prod-core-services02:8200"
disable_mlock = true

storage "raft" {
  path    = "/opt/vault/data"
  
  retry_join {
    leader_tls_servername   = "prod-core-services01"
    leader_api_addr         = "https://prod-core-services01:8200"
    leader_ca_cert_file     = "/etc/step/certs/vault/root_ca.crt"
    leader_client_cert_file = "/etc/step/certs/vault/vault.crt"
    leader_client_key_file  = "/etc/step/certs/vault/vault.key"
  }
  retry_join {
    leader_tls_servername   = "prod-core-services03"
    leader_api_addr         = "https://prod-core-services03:8200"
    leader_ca_cert_file     = "/etc/step/certs/vault/root_ca.crt"
    leader_client_cert_file = "/etc/step/certs/vault/vault.crt"
    leader_client_key_file  = "/etc/step/certs/vault/vault.key"
  }
}

listener "tcp" {
  address            = "0.0.0.0:8200"
  tls_cert_file      = "/etc/step/certs/vault/vault.crt"
  tls_key_file       = "/etc/step/certs/vault/vault.key"
  tls_client_ca_file = "/etc/step/certs/vault/root_ca.crt"
}

service_registration "consul" {
  address      = "http://127.0.0.1:8500"
}
EOF

# Node 3
sudo tee /etc/vault.d/vault.hcl << EOF > /dev/null
ui            = true
cluster_addr  = "https://prod-core-services03:8201"
api_addr      = "https://prod-core-services03:8200"
disable_mlock = true

storage "raft" {
  path    = "/opt/vault/data"
  
  retry_join {
    leader_tls_servername   = "prod-core-services01"
    leader_api_addr         = "https://prod-core-services01:8200"
    leader_ca_cert_file     = "/etc/step/certs/vault/root_ca.crt"
    leader_client_cert_file = "/etc/step/certs/vault/vault.crt"
    leader_client_key_file  = "/etc/step/certs/vault/vault.key"
  }
  retry_join {
    leader_tls_servername   = "prod-core-services02"
    leader_api_addr         = "https://prod-core-services02:8200"
    leader_ca_cert_file     = "/etc/step/certs/vault/root_ca.crt"
    leader_client_cert_file = "/etc/step/certs/vault/vault.crt"
    leader_client_key_file  = "/etc/step/certs/vault/vault.key"
  }
}

listener "tcp" {
  address            = "0.0.0.0:8200"
  tls_cert_file      = "/etc/step/certs/vault/vault.crt"
  tls_key_file       = "/etc/step/certs/vault/vault.key"
  tls_client_ca_file = "/etc/step/certs/vault/root_ca.crt"
}

service_registration "consul" {
  address      = "http://127.0.0.1:8500"
}
EOF
  • To work with vault from the terminal, a few variables are required to be set.
export VAULT_ADDR="https://$(hostname):8200"
export VAULT_CACERT="/etc/step/certs/vault/vault.crt"
export CA_CERT=$(sudo cat /etc/step/certs/vault/certs/root_ca.crt)
export DBUS_SESSION_BUS_ADDRESS=/dev/null
  • Start the Vault service and monitor its startup logs.
sudo systemctl enable vault
sudo systemctl start vault
sudo systemctl status vault
  • Check the vault’s status. Note the Sealed and Unseal Progress lines.
vault status
  • In its current state, the vault is running but sealed or locked. I will require unlock key(s) (something like a password) to unlock it. I’ll initialize the vault to get these unlock key(s). Here I am overridding the default values for recovery-shares and recovery-threshold. For more details, please review the documentation for vault operator init.
vault operator init -recovery-shares=1 -recovery-threshold=1

Note the output of the previous command. It will generate a list of unlock key(s) and a root token. Once unsealed, the root token is used to log into vault. Save this information in a secure location.

  • Now, to unseal the vault, execute the following and enter one of the Vault unseal key(s).
vault operator unseal

  • If required, repeat the prior step with additional unseal keys until the vault is unsealed.

  • If all went well, log into Vault with the root token by pointing your browser to one of the Vault servers (https://:8200/ui/) or by using the terminal.

vault login

Certificate maintenance

As noted earlier when I created the certificates, I opted for 30 days before they need to be renewed. To make renewals easier, let me use the step cli and systemd to automate the process.

  • Install the step cli on all nodes
wget https://dl.smallstep.com/gh-release/cli/gh-release-header/v0.27.2/step-cli_0.27.2_arm64.deb
sudo dpkg -i step-cli_0.27.2_arm64.deb
rm step-cli_0.27.2_arm64.deb
  • Write out a systemd service and timer files for scheduling step
# Service
sudo tee /etc/systemd/system/[email protected] << EOF > /dev/null
[Unit]
Description=Certificate renewer for %I
After=network-online.target
Documentation=https://smallstep.com/docs/step-ca/certificate-authority-server-production
StartLimitIntervalSec=0
PartOf=cert-renewer.target

[Service]
Type=oneshot
User=root

Environment=STEPPATH=/etc/step \
            CERT_LOCATION=/etc/step/certs/%i/%i.crt \
            KEY_LOCATION=/etc/step/certs/%i/%i.key

; ExecCondition checks if the certificate is ready for renewal,
; based on the exit status of the command.
; (In systemd <242, you can use ExecStartPre= here.)
ExecCondition=/usr/bin/step certificate needs-renewal ${CERT_LOCATION}

; ExecStart renews the certificate, if ExecStartPre was successful.
ExecStart=/usr/bin/step ca renew --ca-url https://ca.lab.howto.engineer --force ${CERT_LOCATION} ${KEY_LOCATION}

; Try to reload or restart the systemd service that relies on this cert-renewer
; If the relying service doesn't exist, forge ahead.
; (In systemd <229, use `reload-or-try-restart` instead of `try-reload-or-restart`)
ExecStartPost=/usr/bin/env sh -c "! systemctl --quiet is-active %i.service || systemctl try-reload-or-restart %i"

[Install]
WantedBy=multi-user.target
EOF

# Timer
sudo tee /etc/systemd/system/[email protected] << EOF > /dev/null
[Unit]
Description=Timer for certificate renewal of %I
Documentation=https://smallstep.com/docs/step-ca/certificate-authority-server-production
PartOf=cert-renewer.target

[Timer]
Persistent=true

; Run the timer unit every 15 minutes.
OnCalendar=*:1/15

; Always run the timer on time.
AccuracySec=1us

; Add jitter to prevent a "thundering herd" of simultaneous certificate renewals.
RandomizedDelaySec=5m

[Install]
WantedBy=timers.target
EOF
  • Enable the timer service
sudo systemctl enable --now [email protected]
  • List the active timers
systemctl list-timers | grep cert-renewer

From here on out, step will check every 15 minutes to query the remaining lifetime of the vault certificate. When the default 66% of lifetime has passed, step will start trying to renew the certificate.

Conclusion

In this article, I showed how to:

  • Create certificates using step-ca
  • Installing Hashicorp Vault
  • Configuring Hashicorp Vault using the generated certificates
  • Providing a way to automate certificate renewals

Thanks