Kubernetes Secrets Engine

11min
|
Vault

Challenge

In the DevOps world, some CI/CD pipelines require the ability to manage applications running on a Kubernetes cluster. Granting access to a cluster is a risky scenario, and special care should be taken. Kubernetes Secrets Engine will provide a secure token that gives temporary access to the cluster.

When authenticating a process in Kubernetes, a proof of identity must be presented to the Kubernetes API. For machine users, this is usually a JSON Web Token (JWT) owned by a Kubernetes service account. In Kubernetes, authorization is managed by binding roles to identities. The roles describe the actions an entity is able to perform with the Kubernetes API.

Automated creation of the service role, binding and short lived tokens would be required for an continuous integration pipeline to a Kubernetes cluster. Creation of service accounts is simple enough but the manual process of binding and unbinding is tedious, and becomes a lot to manage.

Solution

Vault's Kubernetes secrets engine manages credentials for customer applications. It generates and manages service account tokens, which in turn have specific capabilities assigned to them. With a configurable TTL, the tokens are automatically revoked once the Vault lease expires. This offers the advantage of only granting what access is needed, when it is needed.

Prerequisites

Vault CLI version 1.11+
Kubernetes command-line interface (CLI)
Minikube
Helm CLI
jwt-cli version 6.0+ - optional, allows you examine fields in JSON Web Tokens.

Set up Vault on Kubernetes

Clone the Learn Vault Kubernetes repo.

$ git clone https://github.com/hashicorp-education/learn-vault-kubernetes.git

Navigate to secrets-engine, and run the rest of this lab in that directory.
```
$ cd learn-vault-kubernetes/secrets-engine
```

Start a Minikube cluster.

$ minikube start
😄  minikube v1.26.1 on Darwin 12.4 (arm64)
✨  Automatically selected the docker driver
📌  Using Docker Desktop driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=4, Memory=10240MB) ...
🐳  Preparing Kubernetes v1.24.3 on Docker 20.10.17 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storage class
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default0

Initialization of Minikube could take a few minutes.

Use Helm to install Vault on Minikube.

$ helm install vault hashicorp/vault -n vault -f values.yaml --create-namespace
NAME: vault
LAST DEPLOYED: Thu Aug 18 09:00:21 2022
NAMESPACE: vault
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Vault!

Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:

https://www.vaultproject.io/docs/

Your release is named vault. To learn more about the release, try:

$ helm status vault
$ helm get manifest vault

Make sure Vault is running.

$ kubectl get pods -n vault
NAME      READY   STATUS    RESTARTS   AGE
vault-0   1/1     Running   0          49s

Allow access to the Vault instance running in your minikube cluster.
```
$ kubectl port-forward -n vault service/vault 8200:8200 1> /dev/null &
```
This command will forward calls through your local 8200 to port 8200 on the Kubernetes Cluster.

Set up local environment variables.

$ export VAULT_DEV_ROOT_TOKEN_ID="root" \
    VAULT_TOKEN="root" \
    VAULT_ADDR="http://127.0.0.1:8200" \
    KUBE_HOST=$(kubectl config view --minify \
        -o 'jsonpath={.clusters[].cluster.server}')

Create a new namespace in Kubernetes.

$ kubectl create namespace demo
namespace/demo created

Now create the role, service accounts and bindings that are used by the Kubernetes secrets engine.

$ kubectl apply -f roles.yaml,serviceAccounts.yaml,bindings.yaml
clusterrole.rbac.authorization.k8s.io/k8s-secrets-abilities created
role.rbac.authorization.k8s.io/demo-role-list-pods created
clusterrole.rbac.authorization.k8s.io/demo-cluster-role-list-pods created
serviceaccount/sample-app created
clusterrolebinding.rbac.authorization.k8s.io/k8s-secrets-abilities-binding created
clusterrolebinding.rbac.authorization.k8s.io/demo-clusterrole-abilities created
rolebinding.rbac.authorization.k8s.io/demo-role-abilities created

Start up an nginx server on yor Minikube.
```
$ kubectl run nginx --image=nginx --namespace demo
pod/nginx created
```
This server will mostly be used to demonstrate the access you will be given through the secretes engine.

Set up Kubernetes Secrets Engine

You will now use the Vault CLI to set up the Kubernetes Secrets Engine.

It will provide a short lived token, that gives you access to the Kubernetes cluster.

Create and initialize Kubernetes secrets engine on vault

$ vault secrets enable kubernetes
Success! Enabled the kubernetes secrets engine at: kubernetes/

This endpoint configures and initializes the plugin with the necessary information to reach the Kubernetes API and authenticate with it.
```
$ vault write -f kubernetes/config
Success! Data written to: kubernetes/config
```

Create a role for the sample app.

$ vault write kubernetes/roles/sample-app \
    service_account_name=sample-app \
    allowed_kubernetes_namespaces=demo \
    token_max_ttl=80h \
    token_default_ttl=1h

Notice that the only allowed namespace is "demo". This roles abilities should be limited to that one namespace.

Create credentials for the sample application in the demo namespace, and receive a service account token in return.

$ vault write kubernetes/creds/sample-app kubernetes_namespace=demo
Key                          Value
---                          -----
lease_id                     kubernetes/creds/sample-app/bvIP7OnOukNz68DJtADydqpe
lease_duration               1h
lease_renewable              false
service_account_name         sample-app
service_account_namespace    demo
service_account_token        eyJhbGciOiJSUzI1NiIsImtpZCI6InFuWWJEaXAtTWxFQzZTbzV5WExoTmNYM3N2bHhTTmdYWGZQcENsbE53N2sifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjYxMTg4MjA3LCJpYXQiOjE2NjExODQ2MDcsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJkZW1vIiwic2VydmljZWFjY291bnQiOnsibmFtZSI6InNhbXBsZS1hcHAiLCJ1aWQiOiI1NWM2NjlhOS02ODJmLTQ2OTEtYjkwNS0yNGMxMWUzNThlYTEifX0sIm5iZiI6MTY2MTE4NDYwNywic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlbW86c2FtcGxlLWFwcCJ9.FSW_lqzQfOApfIbrRzFlqwt9j6vGYlgsGxhftZnPT8_d8HaOXKlK-Gwa7REA9toGvluRAPIRGApbL_INoH2ZcQWCML8GwuzthrkOePiC_Tk30kOEYNlJje2jYLsrKJAOlMpYVZCd-eLf7c_1zrXzYOjk3izD-z8JCEbBW2shVmhWVbsdn4Mf01m9ZExj_J2i98MS9jgnEHJrua4AI4s_pSoP8UzWxCB2eE3GXgqOcI7mRskLuwIo4fI3mmGA0O5Dn3sqIE5T8bOv_t68wWavph3YPYQsUCAqLtKlc0UmrW3p-pkcWg5lAMf0q2zLx0Tz0zoZahNH5qrPTy7P6i-TbQ

Create an environment variable to store the generated service account token.

$ export TEST_TOKEN=<TOKEN>

The export should resemble the following:

$ export TEST_TOKEN=eyJhbGciOiJSUzI1NiIsImtpZCI6ImpsekY0UDMyX1FETFYyOHhDQmFadjZuczNYWlEtbWpyb3lQMlMtNUZmdzAifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjYwNjAwNjU3LCJpYXQiOjE2NjA1OTcwNTcsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJkZW1vIiwic2VydmljZWFjY291bnQiOnsibmFtZSI6InNhbXBsZS1hcHAiLCJ1aWQiOiIzNjIwYTcwMi01YWU1LTRiMjUtYWU2ZS1kZjFkOTNjNTgzNmEifX0sIm5iZiI6MTY2MDU5NzA1Nywic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlbW86c2FtcGxlLWFwcCJ9.AoPJUt9A3EWPtM716gRRzM1e4t-DW86Y5z5HD6T4tguSOVdZ_VEolyEwP_gkPqBJtonf-m-XwQUY4FHw8EfdKXwkP5cJuvQ5T1x1hu9hQyOjl_AaXyI8mTSmOZHlY2PnoeWXs9wqapreDMp91P9rWAqVLsdXZqBl10KfNFS7tPgZ7GmHX15ZphQc1WAeOFwpTbQV1BJOYJAiKZd35gq1J-UEcDzJ5YwppC94ufiWoeGi2f7PzmshYVxdJR0lbFdx7WgtH4BngQ4gfmdQTQRTsECxKtXiqIiI_cyGUdkPo7z-gRV9hsbUcycpf-kdGJsMmBle8hT6L6F0U_siT9zsLQ

Demonstrate token use

Using that token allows you access to the demo namespace.

First, look at the token a bit closer and then use it to list pods in the demo namespace.

Optional Step This will generate another token, but directly return the service_account_token field, and run it through jwt-cli, allowing you to examine the JWT in a bit more detail.

Run this command to examine the payload of the JWT.

$ vault write -field=service_account_token kubernetes/creds/sample-app kubernetes_namespace=demo \
  | jwt decode -

The output is similar too:

Token header
------------
{
  "alg": "RS256",
  "kid": "qnYbDip-MlEC6So5yXLhNcX3svlxSNgXXfPpCllNw7k"
}

Token claims
------------
{
  "aud": [
    "https://kubernetes.default.svc.cluster.local"
  ],
  "exp": 1661188207,
  "iat": 1661184607,
  "iss": "https://kubernetes.default.svc.cluster.local",
  "kubernetes.io": {
    "namespace": "demo",
    "serviceaccount": {
      "name": "sample-app",
      "uid": "55c669a9-682f-4691-b905-24c11e358ea1"
    }
  },
  "nbf": 1661184607,
  "sub": "system:serviceaccount:demo:sample-app"

Notice how the namespace, and app are specified in this token along with timestamps.

Now use cURL, and include the token in the authorization header. This will access the Kubernetes cluster and get a list of pods.

$ curl -k $KUBE_HOST/api/v1/namespaces/demo/pods --header "Authorization: Bearer $TEST_TOKEN"
{
"kind": "PodList",
"apiVersion": "v1",
"metadata": {
  "resourceVersion": "279871"
},
"items": [
  {
    "metadata": {
      "name": "nginx",
      "namespace": "demo",
      "uid": "6673397e-1b77-4758-bf19-ad7429d2ddcf",
      "resourceVersion": "279826",
      "creationTimestamp": "2022-08-22T18:12:20Z",
      "labels": {
        "run": "nginx"

...

The output is long so only a truncated version is given.

Now, that token should not have access to other namespaces. Try the same command but change the demo to default.

$ curl -k $KUBE_HOST/api/v1/namespaces/default/pods --header "Authorization: Bearer $TEST_TOKEN"
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "pods is forbidden: User \"system:serviceaccount:demo:sample-app\" cannot list resource \"pods\" in API group \"\" in the namespace \"default\"",
  "reason": "Forbidden",
  "details": {
    "kind": "pods"
  },
  "code": 403
}

You can create roles as needed. Go ahead and create a new role.

$ vault write kubernetes/roles/existing-role \
    kubernetes_role_name=demo-role-list-pods \
    allowed_kubernetes_namespaces=demo \
    token_max_ttl=72h \
    token_default_ttl=2h

Create a new token and specify the ttl on the command line.
```
$ export TEST_TOKEN=$(vault write -field=service_account_token kubernetes/creds/existing-role kubernetes_namespace=demo ttl=30m)
```
Earlier this was broken into two parts, but now getting the token into the en variable is one command.
Tip
If you are getting the error: The JWT provided is invalid because Error(Base64(InvalidByte(0, 34))) and have VAULT_FORMAT=json then unset the VAULT_FORMAT environment variable.

List pods in the demo namespace.

$ curl -k $KUBE_HOST/api/v1/namespaces/demo/pods --header "Authorization: Bearer $TEST_TOKEN"
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "resourceVersion": "281453"
  },
  "items": [
    {
      "metadata": {
        "name": "nginx",
        "namespace": "demo",
        "uid": "6673397e-1b77-4758-bf19-ad7429d2ddcf",
        "resourceVersion": "279826",
        "creationTimestamp": "2022-08-22T18:12:20Z",
        "labels": {
          "run": "nginx"
        },
...

Response is truncated for brevity's sake.

Revoke those credentials.
```
$ vault lease revoke -prefix kubernetes/creds/existing-role
All revocation operations queued successfully!
```
You can do the same with kubernetes/creds/sample-app if you like, but it had a ttl of 3 hours and will expire soon enough anyways.

Create a cluster role

A Cluster role is not limited to a specific namespace and will have access to the whole Kubernetes cluster. In this section you will create and bind a cluster role, and use it to get details on the Minikube cluster.

Create a new cluster role.

$ vault write kubernetes/roles/rules \
    allowed_kubernetes_namespaces=default \
    allowed_kubernetes_namespaces=demo \
    kubernetes_role_type=ClusterRole \
    token_max_ttl=80h \
    token_default_ttl=1h \
    generated_role_rules='{"rules":[{"apiGroups":[""],"resources":["pods"],"verbs":["list","get"]}]}'

The kubernetes_role_type=ClusterRole specifies that this is a cluster role.

Next, create some credentials for the cluster role.

$ export TEST_TOKEN=$(vault write -field=service_account_token kubernetes/creds/rules kubernetes_namespace=demo cluster_role_binding=true)

Test out the token by listing the pods in the demo namspace.

$ curl -k $KUBE_HOST/api/v1/namespaces/demo/pods --header "Authorization: Bearer $TEST_TOKEN"
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "resourceVersion": "298280"
  },
  "items": [
    {
      "metadata": {
        "name": "nginx",
        "namespace": "demo",
        "uid": "6673397e-1b77-4758-bf19-ad7429d2ddcf",
        "resourceVersion": "279826",
        "creationTimestamp": "2022-08-22T18:12:20Z",
        "labels": {
          "run": "nginx"
        },
...

Check out a different namespace - default. Using a cluster role should give you access to all namspaces.

$ curl -k $KUBE_HOST/api/v1/namespaces/default/pods --header "Authorization: Bearer $TEST_TOKEN"
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "resourceVersion": "298335"
  },
  "items": []
}

You also can look at the Vault namespace.

$ curl -k $KUBE_HOST/api/v1/namespaces/vault/pods --header "Authorization: Bearer $TEST_TOKEN"

Output should be similar to the two examples above.

Switch off the cluster role status.

$ export TEST_TOKEN=$(vault write -field=service_account_token kubernetes/creds/rules kubernetes_namespace=demo cluster_role_binding=false)

No access to view Vault namespace anymore.

$ curl -k $KUBE_HOST/api/v1/namespaces/kube-system/pods --header "Authorization: Bearer $TEST_TOKEN"
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "pods is forbidden: User \"system:serviceaccount:demo:v-token-rules-1661524674-w5ubppfa0edtjewvzwlv4vyu\" cannot list resource \"pods\" in API group \"\" in the namespace \"kube-system\"",
  "reason": "Forbidden",
  "details": {
    "kind": "pods"
  },
  "code": 403
}

Try the demo namespace again.

$ curl -k $KUBE_HOST/api/v1/namespaces/demo/pods --header "Authorization: Bearer $TEST_TOKEN"
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "resourceVersion": "2013"
  },
  "items": [
    {
      "metadata": {
        "name": "nginx",
        "namespace": "demo",
        "uid": "f1313e74-a93c-43f9-addf-963931ec61ab",
        "resourceVersion": "513",
        "creationTimestamp": "2022-08-26T14:05:02Z",
        "labels": {
          "run": "nginx"
        },
...

Go ahead and revoke these credentials.

$ vault lease revoke -prefix kubernetes/creds/rules
All revocation operations queued successfully!

Verify the credentials are revoked.

$ curl -k $KUBE_HOST/api/v1/namespaces/demo/pods --header "Authorization: Bearer $TEST_TOKEN"
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}

Rewrite the rules to give it a bit more power.

$ vault write kubernetes/roles/rules \
    allowed_kubernetes_namespaces=default \
    allowed_kubernetes_namespaces=demo \
    kubernetes_role_type=ClusterRole \
    token_max_ttl=80h \
    token_default_ttl=1h \
    generated_role_rules='{"rules":[{"apiGroups":[""],"resources":["pods"],"verbs":["list","get","delete"]}]}'

Get a new token. This new command will put the token into TEST_TOKEN for you.

$ export TEST_TOKEN=$(vault write -field=service_account_token kubernetes/creds/rules kubernetes_namespace=demo cluster_role_binding=true)

Using that token, delete a pod.

$ curl -k -X DELETE $KUBE_HOST/api/v1/namespaces/demo/pods/nginx --header "Authorization: Bearer $TEST_TOKEN"
{
  "kind": "Pod",
  "apiVersion": "v1",
  "metadata": {
    "name": "nginx",
    "namespace": "demo",
    "uid": "f1313e74-a93c-43f9-addf-963931ec61ab",
    "resourceVersion": "3056",
    "creationTimestamp": "2022-08-26T14:05:02Z",
    "deletionTimestamp": "2022-08-26T15:03:09Z",
    "deletionGracePeriodSeconds": 30,
    "labels": {
      "run": "nginx"
    },
    "managedFields": [
      {
        "manager": "kubectl-run",
        "operation": "Update",
        "apiVersion": "v1",
        "time": "2022-08-26T14:05:02Z",
        "fieldsType": "FieldsV1",
        "fieldsV1": {
          "f:metadata": {
            "f:labels": {
              ".": {},
              "f:run": {}
...

Verify pod deleted.

$ curl -k $KUBE_HOST/api/v1/namespaces/demo/pods/ --header "Authorization: Bearer $TEST_TOKEN"
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "resourceVersion": "391140"
  },
  "items": []
}

With Vault Kubernetes Secrets Engine you can grant temporary access to make major changes to the cluster.

Clean up

The fastest way to ckean up this lab is just to delete Minikube.

$ minikube delete
🔥  Deleting "minikube" in docker ...
🔥  Removing /Users/mrken/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster.

Review

You created a Kubernets Cluster locally using minikube, and added Vault and Nginx to it. Using the Kubernetes Secrets Engine on Vault, recieved an JSON Web Token (JWT) and used it git the API and examine pods in the cluster. These were initially using roles, which are limited to specific namespaces.

Then you created a cluster role, which allows you access the whole cluster. You looked at the pods in a few namespaces and after changing some of the caabilities of the cluster role deleted the Nginx pod.

References

Vault on Kubernetes security considerations

The Vault Secrets Operator on Kubernetes