Using Vault as a CA

As mentioned in the Certificates and trust reference documentation, HashiCorp's Vault can be used to provide either a root or intermediate CA. It can also be deployed HA, as well as provide a secure secrets store which can be used to enable encryption-at-rest for Charmed Kubernetes.

Vault does require an additional database to store its data and (depending on configuration) some manual steps will be required after deployment or any reboot so that secrets, such as certificates and signing keys, can be accessed.

Deploying Charmed Kubernetes with Vault as a root CA

When deploying Charmed Kubernetes manually via the published Juju bundle, it is possible to make use of an overlay file to change the composition and configuration of the cluster.

The following overlay file (download) alters Charmed Kubernetes to use Vault instead of EasyRSA:

applications:
  easyrsa: null
  mysql-innodb-cluster:
    channel: 8.0/stable
    charm: mysql-innodb-cluster
    constraints: cores=2 mem=8G root-disk=64G
    num_units: 3
    options:
      enable-binlogs: true
      innodb-buffer-pool-size: 256M
      max-connections: 2000
      wait-timeout: 3600
  vault:
    channel: 1.7/stable
    charm: vault
    num_units: 1
    options:
      # this makes Vault act as a self-signed root CA
      auto-generate-root-ca-cert: true
  vault-mysql-router:
    channel: 8.0/stable
    charm: mysql-router
relations:
- - etcd:certificates
  - vault:certificates
- - kubeapi-load-balancer:certificates
  - vault:certificates
- - kubernetes-control-plane:certificates
  - vault:certificates
- - kubernetes-worker:certificates
  - vault:certificates
- - mysql-innodb-cluster:db-router
  - vault-mysql-router:db-router
- - vault-mysql-router:shared-db
  - vault:shared-db

Save this to a file named vault-pki-overlay.yaml and deploy with:

juju deploy charmed-kubernetes --overlay ./vault-pki-overlay.yaml

Once the deployment settles, you will notice that several applications are in a blocked state in Juju, with Vault indicating that it needs to be initialised and unsealed. To unseal Vault, you can read the vault charm documentation for in-depth instructions (you may also need to expose Vault), or you can use the Vault client already on the deployed unit with the following steps:

juju ssh vault/0
export HISTCONTROL=ignorespace  # enable leading space to suppress command history
export VAULT_ADDR='http://localhost:8200'
vault operator init -key-shares=5 -key-threshold=3  # outputs 5 keys and a root token
 vault operator unseal {key1}
 vault operator unseal {key2}
 vault operator unseal {key3}
 VAULT_TOKEN={root token} vault token create -ttl 10m  # outputs a {charm token} to auth the charm
exit
juju run vault/0 authorize-charm token={charm token}

Certificate lifespan

By default, the Vault certificates will be valid for one year from creation. To alter this expiration date, you can configure the default-ttl with a new value. E.g. to set new certificates to expire after 720 hours:

juju config vault default-ttl='720h'

New certificates can be generated by running an action on the vault charm:

juju run vault/0 reissue-certificates

For more details on configuring vault certificates, please see the vault charm documentation.

Transitioning an existing cluster from EasyRSA to Vault

An existing Charmed Kubernetes deployment which is using EasyRSA can be transitioned to use Vault as a CA.

Deploy Vault with the recommended database backend:

juju deploy -n 3 mysql-innodb-cluster
juju deploy mysql-router vault-mysql-router
juju deploy vault
juju config vault auto-generate-root-ca-cert=true
juju integrate mysql-innodb-cluster:db-router vault-mysql-router:db-router
juju integrate vault-mysql-router:shared-db vault:shared-db

Unseal Vault as described earlier in this document.

Relate Vault to etcd:

juju integrate vault:certificates etcd:certificates

Wait a few minutes for the cluster to settle, with all units showing as active and idle. Then relate Vault to Kubernetes:

juju integrate vault:certificates kubeapi-load-balancer:certificates
juju integrate vault:certificates kubernetes-control-plane:certificates
juju integrate vault:certificates kubernetes-worker:certificates

Wait a few minutes for the cluster to settle, and ensure that all services and workloads are functioning as expected.

After the transition, you must remove EasyRSA to prevent it from conflicting with Vault:

juju remove-application easyrsa

You will need to re-download the kubectl config file, since it contains the certificate info for connecting to the cluster:

juju ssh kubernetes-control-plane/leader -- cat config > ~/.kube/config

Using Vault as an intermediary CA

If you don't wish Vault to act as a self-signed root CA, you can remove the auto-generate-root-ca-cert: true option from the overlay and follow these instructions to generate a Certificate Signing Request (CSR), have it signed by a trusted root CA, and upload it back to Vault.

Using Vault in Auto-Unseal mode

The Vault charm supports the ability to store and manage the unseal keys and root token using Juju leadership data. Note that this means that the unseal keys can be accessed at any time from the machine that Vault is running on, significantly reducing the security of Vault, particularly with respect to serving as a secure secrets store. If you are comfortable with this reduction in security and don't want to have to deal with the manual steps involved in managing the unseal keys and root token, you can add the following to the options section of vault in the overlay above:

    totally-unsecure-auto-unlock: true

Using Vault with HA

To enable HA for Vault, you will need to first bring up the deployment with Vault in non-HA mode using the instructions above, wait for everything to settle, and then transition Vault to HA mode. This is necessary because Vault requires etcd to be running to enter HA mode, but etcd requires PKI certificates to get up and running, leading to a chicken-and-egg conflict.

Once the deployment is up and settled according to the instructions above, with Vault unsealed and everything functioning, you can then transition Vault to HA mode with the following commands:

juju integrate vault:etcd etcd:db
juju add-unit vault

Once the second unit of Vault is up, you will need to unseal it using the same instructions above with any three of the five unseal keys and the root token you generated previously. For example, assuming vault/1 is the new Vault unit, run the following:

juju ssh vault/1
export HISTCONTROL=ignorespace  # enable leading space to suppress command history
export VAULT_ADDR='http://localhost:8200'
 vault operator unseal {key1}
 vault operator unseal {key2}
 vault operator unseal {key3}
 VAULT_TOKEN={root token} vault token create -ttl 10m  # outputs a new {charm token} to auth the charm
exit
juju run vault/1 authorize-charm token={charm token}

At this point, vault/1 will be running in HA mode with configuration data from etcd. The vault/0 unit will need to be restarted and unsealed again to pick up this new configuration:

juju run vault/0 restart
juju ssh vault/0
export HISTCONTROL=ignorespace  # enable leading space to suppress command history
export VAULT_ADDR='http://localhost:8200'
 vault operator unseal {key1}
 vault operator unseal {key2}
 vault operator unseal {key3}
exit

After unsealing, it may take up to 5 minutes for vault/0 to return to active status. Once it does, verify Vault is in HA mode with one active unit and one standby unit:

juju status vault
  vault/0* ... Unit is ready (active: false, mlock: enabled)
  vault/1 ... Unit is ready (active: true, mlock: enabled)