Manage certificates with ACME clients and the PKI secrets engine

14min
|
Vault
Interactive

HTTPS with TLS is the defacto standard for all web traffic today and production use cases require this level of security at a minimum.

Challenge

Provisioning and maintaining TLS certificate lifecycle for a large number of use cases presents a considerable operational burden. Manually maintaining certificate lifecycle also introduces issues which can result in downtime if certificates are not properly provisioned or renewed on time.

Solution

Starting with version 1.14.0, the Vault PKI secrets engine supports the Automatic Certificate Management Environment (ACME) specification for issuing and renewing leaf server certificates.

You can use ACME-compliant clients with Vault to help automate the leaf server certificate lifecycle.

The hands-on lab scenario presented here uses the Caddy web server's automatic HTTPS functionality with Vault as its ACME server.

Scenario introduction

You'll learn about the PKI secrets engine ACME functionality by deploying and configuring a Caddy web server and a Vault server. First, you'll observe behavior of the Caddy server when not configured to use automatic HTTPS. Then, you'll enable ACME support in a PKI secrets engine instance and configure Caddy to use Vault as its ACME server to enable automatic HTTPS. After configuring the Caddy server, you'll explore the behavior with requests to the Caddy server.

Personas

The end-to-end scenario described in this tutorial involves two personas:

admin with privileged permissions to enable and configure Vault and Caddy.
user makes requests to Caddy.

Prerequisites

Vault CLI installed and in your system PATH.
curl CLI installed and in your system PATH.
Docker installed.
jq installed and in your system PATH for parsing JSON responses.

Versions used for this tutorial

This tutorial was last tested 15 Jun 2023 on macOS using the following software versions.

Operating System

$ sw_vers
ProductName:            macOS
ProductVersion:     13.4
BuildVersion:       22F66

Vault

$ vault version
Vault v1.14.0 (1327dc6728853611f62708e28dbac3ce6cabad1a), built 2023-06-06T18:12:53Z

Caddy

$ docker run caddy caddy version | awk '{print $1}'
v2.6.4

Curl

$ curl --version
curl 7.88.1 (x86_64-apple-darwin22.0) libcurl/7.88.1 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.11 nghttp2/1.51.0
Release-Date: 2023-02-20
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL threadsafe UnixSockets

Docker

$ docker version --format '{{.Server.Version}}'
23.0.5

$ jq --version
jq-1.6

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Lab setup

This hands-on lab uses a Docker container environment consisting of a Vault development mode server container and a Caddy web server container.

Create hands-on lab home

You can create a temporary directory to hold all the content needed for this hands-on lab and then assign its path to an environment variable for later reference.

Open a terminal, and create the directory /tmp/learn-vault-pki.
```
$ mkdir /tmp/learn-vault-pki
```
Export the hands-on directory path as the value to the HC_LEARN_LAB environment variable.
```
$ export HC_LEARN_LAB=/tmp/learn-vault-pki
```

Create a learn-vault Docker network.

$ docker network create \
    --driver=bridge \
    --subnet=10.1.1.0/24 \
    learn-vault

Deploy HTTP Caddy container

Your goal for this section is to pull the Caddy server image and deploy an HTTP based container so that you can explore the server in its most basic configuration.

Pull the latest Caddy server container image.

$ docker pull caddy:2.6.4

Example expected output:

Using default tag: latest
latest: Pulling from library/caddy
91d30c5bc195: Pull complete
7f137c1fd65a: Pull complete
123731571dfc: Pull complete
9ab4cbb8b7b7: Pull complete
Digest:   sha256:ef6ed6e22b469efd5051e1c4cee221d3a0ebebea14bbb5898c8fb4dc70d12d12
Status: Downloaded newer image for caddy:latest
docker.io/library/caddy:latest

Create directories for Caddy configuration and data.

$ mkdir "$HC_LEARN_LAB"/caddy_config "$HC_LEARN_LAB"/caddy_data

Create a simplified index.html file in the hands-on lab home directory.
```
echo "hello world" > "$HC_LEARN_LAB"/index.html
```

Start the Caddy server container.

$ docker run \
    --name caddy-server \
    --hostname caddy-server \
    --network learn-vault \
    --ip 10.1.1.200 \
    --publish 80:80 \
    --volume "$HC_LEARN_LAB"/index.html:/usr/share/caddy/index.html \
    --volume "$HC_LEARN_LAB"/caddy_data:/data \
    --detach \
    --rm \
    caddy:2.6.4

Confirm that the Caddy server container is up.

$ docker ps -f name=caddy-server --format "table {{.Names}}\t{{.Status}}"

Example expected output:

NAMES          STATUS
caddy-server   Up 2 seconds

The Caddy server container was configured to publish the HTTP port to localhost. Use curl to make an HTTP request to the Caddy server.
```
$ curl http://localhost/
```
Example expected output:
```
hello world
```
Your HTTP request was successful and the response includes hello world.
You have not yet configured the Caddy server to use its automatic HTTPS feature, so an HTTPS request is expected to fail. Try an HTTPS request to observe the result.
```
$ curl https://localhost/
```
Example expected output:
```
curl: (7) Failed to connect to localhost port 443 after 12 ms: Couldn't connect to server
```
Now that you've explored the Caddy server behavior without HTTPS, go ahead and stop the Caddy container and proceed to deploying the Vault container.
```
$ docker stop caddy-server
caddy-server
```

Deploy Vault container

Your goal for this section is to deploy a dev mode Vault server container that you'll use for the hands-on lab.

Note

The dev mode server does not support TLS for non-loopback addresses, and is used without TLS just for this tutorial. Vault should always be used with TLS in production deployments. This configuration requires a certificate file and key file on each Vault host.

Pull the latest Vault Docker image.

$ docker pull hashicorp/vault:latest

Example abbreviated output:

latest: Pulling from hashicorp/vault
...snip...
docker.io/hashicorp/vault:latest

Start Vault server container.

$ docker run \
      --name=learn-vault \
      --hostname=learn-vault \
      --network=learn-vault \
      --add-host caddy-server.learn.internal:10.1.1.200 \
      --ip 10.1.1.100 \
      --publish 8200:8200 \
      --env VAULT_ADDR="http://localhost:8200" \
      --env VAULT_CLUSTER_ADDR="http://learn-vault:8201" \
      --env VAULT_API_ADDR="http://learn-vault:8200" \
      --cap-add=IPC_LOCK \
      --detach \
      --rm \
      hashicorp/vault vault server -dev -dev-listen-address 0.0.0.0:8200 -dev-root-token-id root

Note

The dev mode server uses an initial root token value that is the literal string root. This is used just for the purposes simplifying steps in the tutorial.

Confirm that the Vault server container is up.

$ docker ps -f name=learn-vault --format "table {{.Names}}\t{{.Status}}"

Example expected output:

NAMES         STATUS
learn-vault   Up 5 seconds

Export an environment variable for the vault CLI to address the Vault server.
```
$ export VAULT_ADDR=http://127.0.0.1:8200
```

Verify Vault server status.

$ vault status

Example expected output:

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.14.0
Build Date      2023-06-06T18:12:53Z
Storage Type    inmem
Cluster Name    vault-cluster-b8904427
Cluster ID      a062fe3d-79d6-2916-e59b-d072f457dbf2
HA Enabled      false

Authenticate to Vault with the initial root token.
```
$ vault login -no-print root
```

The Vault server is now ready for you to enable and configure the PKI secrets engine.

Enable PKI secrets engine

Your goal for this section is to enable PKI infrastructure consisting of 2 PKI secrets engine instances representing a root CA (pki) and intermediate CA (pki_int) for the learn.internal domain.

This configuration is based on steps like those in the Build Your Own Certificate Authority (CA) tutorial. You are encouraged to complete the hands-on lab in that tutorial if you are unfamiliar with the PKI secrets engine.

To save time and typing, you can use this example shell script to enable and configure the secrets engines for the hands-on lab.

Use a text editor to write and save this content to the file $HC_LEARN_LAB/pki/enable_engines.sh. You may need to first create the pki folder.

enable_engines.sh

#!/bin/bash

set -euxo pipefail

vault secrets enable pki
vault secrets tune -max-lease-ttl=87600h pki
vault write -field=certificate pki/root/generate/internal \
   common_name="learn.internal" \
   issuer_name="root-2023" \
   ttl=87600h > root_2023_ca.crt
vault write pki/config/cluster \
   path=http://10.1.1.100:8200/v1/pki \
   aia_path=http://10.1.1.100:8200/v1/pki
vault write pki/roles/2023-servers \
   allow_any_name=true \
   no_store=false
vault write pki/config/urls \
   issuing_certificates={{cluster_aia_path}}/issuer/{{issuer_id}}/der \
   crl_distribution_points={{cluster_aia_path}}/issuer/{{issuer_id}}/crl/der \
   ocsp_servers={{cluster_path}}/ocsp \
   enable_templating=true
vault secrets enable -path=pki_int pki
vault secrets tune -max-lease-ttl=43800h pki_int
vault write -format=json pki_int/intermediate/generate/internal \
   common_name="learn.internal Intermediate Authority" \
   issuer_name="learn-intermediate" \
   | jq -r '.data.csr' > pki_intermediate.csr
vault write -format=json pki/root/sign-intermediate \
   issuer_ref="root-2023" \
   csr=@pki_intermediate.csr \
   format=pem_bundle ttl="43800h" \
   | jq -r '.data.certificate' > intermediate.cert.pem
vault write pki_int/intermediate/set-signed certificate=@intermediate.cert.pem
vault write pki_int/config/cluster \
   path=http://10.1.1.100:8200/v1/pki_int \
   aia_path=http://10.1.1.100:8200/v1/pki_int
vault write pki_int/roles/learn \
   issuer_ref="$(vault read -field=default pki_int/config/issuers)" \
   allow_any_name=true \
   max_ttl="720h" \
   no_store=false
vault write pki_int/config/urls \
   issuing_certificates={{cluster_aia_path}}/issuer/{{issuer_id}}/der \
   crl_distribution_points={{cluster_aia_path}}/issuer/{{issuer_id}}/crl/der \
   ocsp_servers={{cluster_path}}/ocsp \
   enable_templating=true

Note

The values used here are for just the hands-on lab. You should change the values of common_name, issuer_name, path, and aia_path to match your own values when using these commands outside of the context of the hands-on lab.

Make the script executable.

$ chmod +x $HC_LEARN_LAB/pki/enable_engines.sh

Execute the script to enable and configure the secrets engines from within the pki folder.

$ ./enable_engines.sh

Abbreviated expected output:

Success! Enabled the pki secrets engine at: pki/
Success! Tuned the secrets engine at: pki/
Key                                   Value
---                                   -----
allow_any_name                        true
...snip...
Key                        Value
---                        -----
crl_distribution_points    [{{cluster_aia_path}}/issuer/{{issuer_id}}/crl/der]
enable_templating          true
issuing_certificates       [{{cluster_aia_path}}/issuer/{{issuer_id}}/der]
ocsp_servers               [{{cluster_path}}/ocsp]

List the enabled PKI secrets engines.

$ vault secrets list

Example expected output:

Path          Type         Accessor              Description
----          ----         --------              -----------
cubbyhole/    cubbyhole    cubbyhole_1cc418b5    per-token private secret storage
identity/     identity     identity_ddedf595     identity store
pki/          pki          pki_e6a55aa7          n/a
pki_int/      pki          pki_bee5d418          n/a
secret/       kv           kv_f5b4b89f           key/value secret storage
sys/          system       system_17702eb2       system endpoints used for control, policy and debugging

The PKI secrets engines are enabled with a basic configuration.

Configure ACME

Now that you've set up a basic PKI infrastructure, you can enable and configure ACME on the intermediate CA mounted at pki_int with 2 additional steps.

Ensure that the config/cluster configuration is present and correct, or ACME won't work.
```
$ vault read pki_int/config/cluster
```
Example expected output:
```
Key         Value
---         -----
aia_path    http://10.1.1.100:8200/v1/pki_int
path        http://10.1.1.100:8200/v1/pki_int
```
Note
This configuration must be set on all clusters participating in Enterprise Performance Replication.

Tune the intermediate CA PKI mount at pki_int to enable the ACME required headers.

$ vault secrets tune \
      -passthrough-request-headers=If-Modified-Since \
      -allowed-response-headers=Last-Modified \
      -allowed-response-headers=Location \
      -allowed-response-headers=Replay-Nonce \
      -allowed-response-headers=Link \
      pki_int

Example expected output:

Success! Tuned the secrets engine at: pki_int/

Then, enable ACME functionality on the intermediate CA.

$ vault write pki_int/config/acme enabled=true

Example expected output:

Key                         Value
---                         -----
allowed_issuers             [*]
allowed_roles               [*]
default_directory_policy    sign-verbatim
dns_resolver                n/a
eab_policy                  not-required
enabled                     true

Ensure that the config/cluster configuration is present and correct, or ACME won't work.

$ curl \
    --silent \
    --header "X-Vault-Token: root" \
    --request GET \
    $VAULT_ADDR/v1/pki_int/config/cluster \
    | jq

Example expected output:

{
"request_id": "83344e57-421c-0c19-3374-bfa546743f69",
"lease_id": "",
"renewable": false,
"lease_duration": 0,
"data": {
   "aia_path": "http://10.1.1.100:8200/v1/pki_int",
   "path": "http://10.1.1.100:8200/v1/pki_int"
},
"wrap_info": null,
"warnings": null,
"auth": null
}

Note

This configuration must be set on all clusters participating in Enterprise Performance Replication.

Prepare a payload you'll use to tune the intermediate CA PKI mount at pki_int, and enable the ACME required headers.

$ cat > "$HC_LEARN_LAB"/tune-secrets-engine.json << EOF
{
    "passthrough_request_headers": "If-Modified-Since",
    "allowed_response_headers": ["Last-Modified","Location","Replay-Nonce","Link"]
}
EOF

Tune the intermediate CA PKI mount.

$ curl \
 --header "X-Vault-Token: root" \
 --request POST \
 --data @"$HC_LEARN_LAB"/tune-secrets-engine.json \
 $VAULT_ADDR/v1/sys/mounts/pki_int/tune

This command is expected to produce no output.

Enable ACME on the intermediate CA.

$ curl \
    --silent \
    --header "X-Vault-Token: root" \
    --request POST \
    --data '{"enabled": true}' \
    $VAULT_ADDR/v1/pki_int/config/acme \
    | jq

Example expected output:

{
"request_id": "3ad5317e-0697-a037-1217-180b1878fee1",
"lease_id": "",
"renewable": false,
"lease_duration": 0,
"data": {
   "allowed_issuers": [
      "*"
   ],
   "allowed_roles": [
      "*"
   ],
   "default_directory_policy": "sign-verbatim",
   "dns_resolver": "",
   "eab_policy": "not-required",
   "enabled": true
},
"wrap_info": null,
"warnings": null,
"auth": null
}

Access the Vault UI at http://localhost:8200
Enter the initial root token value root into the Token field.
Click Sign in.

Enter the following into the terminal text input and press return.

$ vault write /sys/mounts/pki_int/tune \
    passthrough_request_headers="If-Modified-Since" \
    allowed_response_headers="Last-Modified,Location,Replay-Nonce,Link"

Vault web UI console commands

Close the terminal by clicking the X.
In the list of enabled secrets engines, click pki_int.
In the top navigation, click Configuration.
Under ACME Config, click the checkbox for Acme enabled.
No further configuration is necessary for this example, so you can scroll to the bottom of the page, and click Save.

You have enabled and configured ACME on the PKI secrets engine intermediate CA.

Redeploy Caddy container with HTTPS

Your goal for this section is to deploy a Caddy server container configured to use its automatic HTTPS feature to fetch and use its TLS certificate from Vault.

Caddy is a stateful server and stores its state in $HC_LEARN_LAB/caddy_data/caddy; remove the earlier Caddy server's state.
```
$ rm -rf $HC_LEARN_LAB/caddy_data/caddy
```
Create a Caddyfile to configure the Caddy server. It specifies that Caddy listens on TCP port 443, sets a content root, enables the basic static file server, and defines the Vault server as its ACME CA.
```
$ cat > "$HC_LEARN_LAB"/caddy_config/Caddyfile << EOF
   {
      acme_ca http://10.1.1.100:8200/v1/pki_int/acme/directory
   }
```

  caddy-server {
     root * /usr/share/caddy
     file_server browse
  }

EOF


1. Now, start the Caddy server container.

```shell-session
$ docker run \
    --name caddy-server \
    --hostname caddy-server \
    --network learn-vault \
    --ip 10.1.1.200 \
    --publish 443:443 \
    --volume "$HC_LEARN_LAB"/caddy_config/Caddyfile:/etc/caddy/Caddyfile \
    --volume "$HC_LEARN_LAB"/index.html:/usr/share/caddy/index.html \
    --volume "$HC_LEARN_LAB"/caddy_data:/data \
    --detach \
    --rm \
    caddy:2.6.4

Confirm that the Caddy server container is up.

$ docker ps -f name=caddy-server --format "table {{.Names}}\t{{.Status}}"

Example expected output:

NAMES          STATUS
caddy-server   Up 2 seconds

Make an HTTPS request to Caddy

You can now try a request to the HTTPS enabled Caddy server with curl. You'll need to specify the root CA certificate so that curl can validate the certificate chain.

$ curl \
    --cacert "$HC_LEARN_LAB"/pki/root_2023_ca.crt \
    --resolve caddy-server:443:127.0.0.1 \
    https://caddy-server

Example expected output:

hello world

A successful response indicates that Caddy is now using automatic HTTPS with Vault as its ACME CA.

If you get errors such as SSL routines:ST_CONNECT:tlsv1 alert internal error check that the host name used to test connectivity (in these examples caddy-server) and the certificate's common name or subject alternative names are an exact match. If you are using openssl s_client, specify the -servername option to send the Server Name Indication (SNI) extension matching the domain name on the certificate.

Cleanup

In a terminal session, stop the Caddy server; the container is automatically removed by Docker.
```
$ docker stop caddy-server
```
Stop the Vault server; the container is automatically removed by Docker.
```
$ docker stop learn-vault
```
Remove the Docker network.
```
$ docker network rm learn-vault
```
Change to your home directory.
```
cd
```
Remove the hands-on lab temporary directory.
```
$ rm -rf "$HC_LEARN_LAB"
```

Summary

You've learned how to configure the PKI secrets engine to enable ACME, and manage the lifecycle of a Caddy server TLS certificate with Vault.

Next steps

Now that you have an understanding of the basics around ACME with the PKI Secrets engine, you are encouraged to review the Automate Rotation with ACME section of the API documentation. This section contains important notes and caveats, which you should fully understand before implementing ACME with Vault in your use case.

You can also continue your PKI secrets engine learning journey by exploring the Build Certificate Authority (CA) in Vault with an offline Root and PKI Unified CRL and OCSP With Cross Cluster Revocation tutorials.

Help and reference

Vault Certificate Authority and offline root

PKI Unified CRL and OCSP