HashiCorp Developer AI Private beta now available Sign up today
Nomad
Vault

Generate mTLS certificates for Nomad using Vault

  • 12min
  • |
  • Vault
  • Nomad

You can use Vault's PKI Secrets Engine to generate and renew dynamic X.509 certificates for your Nomad cluster nodes and Vault Agent to automatically create the appropriate certificate and key files on your nodes. With this method, each node has a unique certificate with a relatively short time-to-live (TTL). This feature, along with automatic certificate rotation, allows you to safely and securely scale your cluster while using mutual TLS (mTLS).

In this tutorial, you will secure your existing Nomad cluster with mTLS using Vault's PKI secrets engine to create both a root and intermediate CA. You will also use Vault Agent to fetch, renew, and periodically rotate your mTLS certificates on your Nomad nodes.

Prerequisites

Running this tutorial requires the following:

  • A Nomad environment with Vault installed. You can use this repository to provision a sandbox environment. This tutorial will assume a cluster with one server node and three client nodes.

Note

This tutorial is for demonstration purposes and is using a single Nomad server with a Vault server configured alongside it. In a production cluster, we recommend 3 or 5 Nomad server nodes along with a separate Vault cluster. Refer to the Vault Reference Architecture page to learn how to securely deploy a Vault cluster.

Prepare Vault

If you already have a Vault environment, skip ahead to Log in to Vault.

Initialize Vault server

Run the following command to initialize the Vault server and receive an unseal key and initial root token. If you are running the environment provided in this guide, the Vault server is co-located with the Nomad server. Be sure to note the unseal key and initial root token as you will need these two pieces of information.

$ vault operator init -key-shares=1 -key-threshold=1

The vault operator init command creates a single Vault unseal key for convenience. For a production environment, we recommended that you create at least five unseal key shares and securely distribute them to independent operators. The command defaults to five key shares and a key threshold of three. If you have provisioned more than one server, the others will become standby nodes and remain unsealed.

Unseal Vault

Run the command and provide your unseal key.

$ vault operator unseal

The output looks similar to the following.

Key                    Value
---                    -----
Seal Type              shamir
Initialized            true
Sealed                 false
Total Shares           1
Threshold              1
Version                1.0.3
Cluster Name           vault-cluster-d1b6513f
Cluster ID             87d6d13f-4b92-60ce-1f70-41a66412b0f1
HA Enabled             true
HA Cluster             n/a
HA Mode                standby
Active Node Address    <none>

Log in to Vault

Use the login command to authenticate with Vault using the initial root token from earlier.

$ vault login
Token (will be hidden): <your initial root token>

Successful output looks similar to the following.

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
...

Prepare the PKI environment

This tutorial uses a common and recommended pattern which is to have one mount act as the root CA and the other as the intermediate CA. The root CA signs only the intermediate CA CSRs from other PKI secrets engines. For a higher level of security, you can store your CA outside of Vault and use the PKI engine only as an intermediate CA.

Generate the root CA

Enable the root PKI secrets engine at the pki path.

$ vault secrets enable pki
Success! Enabled the pki secrets engine at: pki/

Tune the PKI secrets engine to issue certificates with a maximum time-to-live (TTL) of 87600 hours.

$ vault secrets tune -max-lease-ttl=87600h pki
Success! Tuned the secrets engine at: pki/

Generate the root certificate and save the certificate as CA_cert.crt.

Note

If you already have an existing root CA certificate in your organization, you can use that in place of generating a new one.

$ vault write -field=certificate pki/root/generate/internal \
    common_name="global.nomad" ttl=87600h > CA_cert.crt

Generate the intermediate CA and CSR

Enable the intermediate PKI secrets engine at the pki_int path.

$ vault secrets enable -path=pki_int pki
Success! Enabled the pki secrets engine at: pki_int/

Tune the PKI secrets engine at the pki_int path to issue certificates with a maximum time-to-live (TTL) of 43800 hours.

$ vault secrets tune -max-lease-ttl=43800h pki_int
Success! Tuned the secrets engine at: pki_int/

Generate a CSR from your intermediate CA and save it as pki_intermediate.csr.

$ vault write -format=json pki_int/intermediate/generate/internal \
    common_name="global.nomad Intermediate Authority" \
    ttl="43800h" | jq -r '.data.csr' > pki_intermediate.csr

Sign and deploy the intermediate CA certificate

Sign the intermediate CA CSR with the root certificate and save it as intermediate.cert.pem.

$ vault write -format=json pki/root/sign-intermediate \
    csr=@pki_intermediate.csr format=pem_bundle \
    ttl="43800h" | jq -r '.data.certificate' > intermediate.cert.pem

Import the signed intermediate certificate into Vault.

$ vault write pki_int/intermediate/set-signed certificate=@intermediate.cert.pem

Set up the AppRole auth method

Vault offers many authentication methods including AWS, GCP, Azure, GitHub, LDAP, and Okta. In a production environment running in a cloud provider, we recommend using the machine's identity and the cloud provider's auth method.

The AppRole authentication method allows applications or services to authenticate with Vault-defined roles. For on-premises environments, we recommend AppRoles. In this tutorial, Nomad is using the AppRole auth method to retrieve certificates from Vault with the PKI engine.

Enable the AppRole auth method.

$ vault auth enable approle
Success! Enabled approle auth method at: approle/

Create a policy to access the role endpoint

Create a policy file named tls-policy.hcl, add the following contents to it, and save the file. This adds only the update capability for the pki_int/issue/nomad-cluster path. Refer to the Vault policies page for additional information.

tls-policy.hcl
path "pki_int/issue/nomad-cluster" {
  capabilities = ["update"]
}

Write the policy into Vault.

$ vault policy write tls-policy tls-policy.hcl
Success! Uploaded policy: tls-policy

Create an approle

Create an approle named nomad-cluster. Consider increasing or decreasing the TTL and SecretID values to match the your environment.

$ vault write auth/approle/role/nomad-cluster \
  token_type=batch \
  token_policies="tls-policy" \
  secret_id_ttl=10m \
  token_ttl=10m \
  token_max_ttl=15m \
  secret_id_num_uses=10

The output looks similar to the following.

Success! Data written to: auth/approle/role/nomad-cluster

Next, get the role ID and save it to a file.

$ vault read -field="role_id" auth/approle/role/nomad-cluster/role-id > roleid

Then, get the SecretID and save it to a file. The SecretID is similar to a password in that you should keep it secret and confidential. Refer to the best practices page for additional information about working with the SecretID.

$ vault write -field="secret_id" auth/approle/role/nomad-cluster/secret-id > secretid

Create and populate the templates directory

The templates directory contains the files used by Vault Agent to render the certificates and keys on the nodes in your cluster.

Create a directory named templates in /opt/nomad.

$ sudo mkdir /opt/nomad/templates

Create a template file for each configuration below, add the contents to it, and save it with the filename given.

We recommend creating only the server and CLI files for server nodes and client and CLI files for client nodes.

Note

The common_name value in the files below must match the Nomad node names in your cluster which include the node name and region. If you have a different configuration for the nodes in your cluster, be sure to update the values.

/opt/nomad/templates/server.agent.crt.tpl
{{ with pkiCert "pki_int/issue/nomad-cluster" "common_name=server.global.nomad" "ttl=24h" "alt_names=localhost" "ip_sans=127.0.0.1"}}
{{ .Data.Cert }}
{{ end }}

/opt/nomad/templates/server.agent.key.tpl
{{ with pkiCert "pki_int/issue/nomad-cluster" "common_name=server.global.nomad" "ttl=24h" "alt_names=localhost" "ip_sans=127.0.0.1"}}
{{ .Data.Key }}
{{ end }}

/opt/nomad/templates/server.ca.crt.tpl
{{ with pkiCert "pki_int/issue/nomad-cluster" "common_name=server.global.nomad" "ttl=24h"}}
{{ .Data.CA }}
{{ end }}

Create the certificates output directory

Create the rendered-cert-files directory to contain the generated certificate files.

$ sudo mkdir /opt/nomad/rendered-cert-files

Start Vault Agent

Create a directory for the configuration named vault-agent in /opt/vault.

$ sudo mkdir /opt/vault/vault-agent

Create the Vault Agent configuration file, name it vault_agent.conf, add the following contents to it, and save the file in the /opt/vault/vault-agent directory.

Modify this configuration file to match the templates created for each specific node from the previous step. Include or remove the server and client template blocks as necessary.

/opt/vault/vault-agent/vault_agent.conf
pid_file = "./pidfile"

vault {
  address = "https://127.0.0.1:8200"

  # For demonstration purposes only. In a production environment,
  # tls_skip_verify value should NOT be set to true.
  tls_skip_verify = true
}

auto_auth {
  method {
    type      = "approle"

    config = {
      role_id_file_path = "/opt/vault/vault-agent/roleid"
      secret_id_file_path = "/opt/vault/vault-agent/secretid"
      remove_secret_id_file_after_reading = false
    }
  }

  sink {
    type = "file"
    wrap_ttl = "30m"
    config = {
      path = "sink_file_wrapped_1.txt"
    }
  }

  sink {
    type = "file"
    config = {
      path = "sink_file_unwrapped_2.txt"
    }
  }
}

listener "tcp" {
  address = "127.0.0.1:8100"
  tls_disable = true
}

template_config {
  static_secret_render_interval = "10m"
  exit_on_retry_failure = true
  max_connections_per_host = 20
}

# Server templates; remove if on client node

template {
  source       = "/opt/nomad/templates/server.agent.crt.tpl"
  destination  = "/opt/nomad/rendered-cert-files/server.agent.crt"
}

template {
  source       = "/opt/nomad/templates/server.agent.key.tpl"
  destination  = "/opt/nomad/rendered-cert-files/server.agent.key"
}

template {
  source       = "/opt/nomad/templates/server.ca.crt.tpl"
  destination  = "/opt/nomad/rendered-cert-files/server.ca.crt"
}

# Client templates; remove if on server node

template {
  source       = "/opt/nomad/templates/client.agent.crt.tpl"
  destination  = "/opt/nomad/rendered-cert-files/client.agent.crt"
}

template {
  source       = "/opt/nomad/templates/client.agent.key.tpl"
  destination  = "/opt/nomad/rendered-cert-files/client.agent.key"
}

template {
  source       = "/opt/nomad/templates/client.ca.crt.tpl"
  destination  = "/opt/nomad/rendered-cert-files/client.ca.crt"
}

# CLI templates

template {
  source       = "/opt/nomad/templates/cli.crt.tpl"
  destination  = "/opt/nomad/rendered-cert-files/cli.crt"
}

template {
  source       = "/opt/nomad/templates/cli.key.tpl"
  destination  = "/opt/nomad/rendered-cert-files/cli.key"
}

Then, start Vault Agent on each node.

$ sudo vault agent -config=/opt/vault/vault-agent/vault_agent.conf

The output shows the agent rendering the local files with the certificate data.

# ...
2024-11-26T10:09:54.347-0500 [INFO]  agent: (runner) rendered "/opt/nomad/templates/cli.crt.tpl" => "/opt/nomad/rendered-cert-files/cli.crt"
2024-11-26T10:09:54.367-0500 [INFO]  agent: (runner) rendered "/opt/nomad/templates/server.ca.crt.tpl" => "/opt/nomad/rendered-cert-files/server.ca.crt"
2024-11-26T10:09:54.405-0500 [INFO]  agent: (runner) rendered "/opt/nomad/templates/client.agent.key.tpl" => "/opt/nomad/rendered-cert-files/client.agent.key"
2024-11-26T10:09:54.425-0500 [INFO]  agent: (runner) rendered "/opt/nomad/templates/server.agent.key.tpl" => "/opt/nomad/rendered-cert-files/server.agent.key"
2024-11-26T10:09:54.463-0500 [INFO]  agent: (runner) rendered "/opt/nomad/templates/cli.key.tpl" => "/opt/nomad/rendered-cert-files/cli.key"
2024-11-26T10:09:54.484-0500 [INFO]  agent: (runner) rendered "/opt/nomad/templates/client.agent.crt.tpl" => "/opt/nomad/rendered-cert-files/client.agent.crt"
2024-11-26T10:09:54.569-0500 [INFO]  agent: (runner) rendered "/opt/nomad/templates/server.agent.crt.tpl" => "/opt/nomad/rendered-cert-files/server.agent.crt"
2024-11-26T10:09:54.592-0500 [INFO]  agent: (runner) rendered "/opt/nomad/templates/client.ca.crt.tpl" => "/opt/nomad/rendered-cert-files/client.ca.crt"

Configure Nomad to use TLS

Add the following tls stanza to the agent configuration file of each Nomad server and client node in the cluster.

/etc/nomad.d/nomad.hcl
tls {
  http = true
  rpc  = true
  rpc_upgrade_mode = true

  ca_file   = "/opt/nomad/rendered-cert-files/server.ca.crt"
  cert_file = "/opt/nomad/rendered-cert-files/server.agent.crt"
  key_file  = "/opt/nomad/rendered-cert-files/server.agent.key"

  verify_server_hostname = true
  verify_https_client    = true
}

The rpc_upgrade_mode attribute ensures that the Nomad servers accept both TLS and non-TLS connections during the upgrade.

Reload the Nomad configuration on each node.

$ systemctl reload nomad

After the reload, remove the rpc_upgrade_mode = true line from each server agent configuration file and save the file. This instructs the servers to only accept TLS connections.

/etc/nomad.d/nomad.hcl
 tls {
   http = true
   rpc  = true
-  rpc_upgrade_mode = true
 
   # ...

Then, reload the Nomad configuration on each server node again. Refer to the RPC Upgrade Mode page for additional information.

$ systemctl reload nomad

Configure the Nomad CLI

The Nomad CLI defaults to HTTP instead of HTTPS. Configure the Nomad CLI to connect using TLS by providing paths to the certificate files through environment variables.

export NOMAD_CACERT="/opt/nomad/rendered-cert-files/server.ca.crt"
export NOMAD_CLIENT_CERT="/opt/nomad/rendered-cert-files/cli.crt"
export NOMAD_CLIENT_KEY="/opt/nomad/rendered-cert-files/cli.key"

Run the status command to confirm connectivity from the CLI.

$ nomad status
No running jobs

Encrypt server gossip

All communications over RPC and HTTP in your Nomad cluster are now secure with mTLS.

Nomad servers also communicate with the Serf gossip protocol, which does not use TLS. To learn how to configure gossip encryption, follow the Enable Gossip Encryption for Nomad tutorial.

Next steps

In this tutorial, you configured the Vault PKI engine, installed and used Vault Agent to automatically create certificate files on cluster nodes, and enabled mTLS with those certificates to encrypt traffic between the nodes.

Continue your learning with these resources:

Previous

Vault token generation

 Next

Dynamic secrets

This tutorial also appears in:

  • 11 tutorials
    HashiCorp product integrations
    Vault can manage secrets associated with other HashiCorp products.
    • Vault