Init container to authenticate with Vault from within a Kubernetes namespace
e45fcfb9
9 MB
2 months ago
33M
Name
Layer
Last update
6 months ago
6 months ago
6 months ago
6 months ago
6 months ago
6 months ago
6 months ago
6 months ago
6 months ago
6 months ago
6 months ago
6 months ago
6 months ago
2 months ago
Readme

Go Report Card Docker Build

Table of Contents generated with DocToc

Credits

Based on the work of Seth Vargo

Scenarios

Scenario 1 - Get a Vault token for one time use

Start the Init Container vault-kubernetes-authenticator to authenticate to Vault and get a Vault token.

The Vault token will expire after the given TTL.

Scenario 2 - Sync Vault secrets to Kubernetes secrets

Start the Init Container vault-kubernetes-authenticator to authenticate to Vault and get a Vault token.

After successful completion, start the Init Container vault-kubernetes-synchronizer to synchronize secrets to Kubernetes.

The Vault token will expire after the given TTL.

Scenario 3 - Get a Vault token for use during the lifetime of a pod

Start the Init Container vault-kubernetes-authenticator to authenticate to Vault and get a Vault token.

After successful completion start the Sidecar Container vault-kubernetes-token-renewer to regularly renew your Vault token.

Issues

vault-kubernetes-token-renewer container will be restarted if the token renewal fails (for restartPolicy=always). When the token cannot be renewed (e.g. the token is in the meantime expired):

  • let the pod terminate and restart. On restart vault-kubernetes-authenticator will issue a new token. A possible solution could be to use Share Process Namespace between Containers in a Pod (Kubernetes 1.12 beta) and Container Lifecycle Hooks
  • let vault-kubernetes-token-renewer re-authenticate and update VAULT_TOKEN_PATH if the token in VAULT_TOKEN_PATH is invalid. The token consumer needs to observe VAULT_TOKEN_PATH for changes (inotify) or read VAULT_TOKEN_PATH on every connect to Vault (isn't a thing because VAULT_TOKEN_PATH is usually in-memory). This can be done independent from the previous case because the token will be valid on after pod creation

removed go.sum from repo due to issue with go version and k8s.io/client-go:

go: verifying k8s.io/client-go@v9.0.0+incompatible: checksum mismatch

Vault client configuration

The usual environment variables for Vault will be used:

  • VAULT_ADDR
  • VAULT_CACERT
  • VAULT_CAPATH
  • VAULT_CLIENT_CERT
  • VAULT_CLIENT_KEY
  • VAULT_CLIENT_TIMEOUT
  • VAULT_SKIP_VERIFY
  • VAULT_TLS_SERVER_NAME
  • VAULT_WRAP_TTL
  • VAULT_MAX_RETRIES
  • VAULT_TOKEN
  • VAULT_MFA
  • VAULT_RATE_LIMIT

see https://godoc.org/github.com/hashicorp/vault/api#Config.ReadEnvironment

the minimal configuration is VAULT_ADDR with VAULT_SKIP_VERIFY=true

Init Container vault-kubernetes-authenticator

Configuration

  • VAULT_ROLE - Required the name of the Vault role to use for authentication.

  • VAULT_TOKEN_PATH - the destination path on disk to store the token. Usually this is a shared volume.

  • VAULT_AUTH_MOUNT_PATH - the name of the mount where the Kubernetes auth method is enabled. This defaults to kubernetes, but if you changed the mount path you will need to set this value to that path (vault auth enable -path=k8s kubernetes -> VAULT_AUTH_MOUNT_PATH=k8s)

  • SERVICE_ACCOUNT_TOKEN_PATH - the path on disk where the Kubernetes service account jtw token lives. This defaults to /var/run/secrets/kubernetes.io/serviceaccount/token.

  • ALLOW_FAIL - the container will successfully terminate even if the authentication to Vault failed, no token will be written to VAULT_TOKEN_PATH. This condition needs to be handeled in the succeeding container. (default: "false")

Example

$ k logs vault-kubernetes-authenticator-5675d58d95-4wd8v -c vault-kubernetes-authenticator
2018/11/26 14:56:29 successfully authenticated to vault
2018/11/26 14:56:29 successfully stored vault token at /home/vault/.vault-token

$ k exec -ti vault-kubernetes-authenticator-5675d58d95-4wd8v sh
~ $ VAULT_TOKEN=$(cat /home/vault/.vault-token)
~ $ echo $VAULT_TOKEN
8Pj0EzFLWQv8uWcjbP9hF1MB
~ $

Init Container vault-kubernetes-synchronizer

Depends on Init Container vault-kubernetes-authenticator

  • each Kubernetes secrets created by vault-kubernetes-synchronizer gets the annotation vault-secret: <vault secret path>

  • Existing labels are retained and configured labels are appended to existing ones. If an existing label has the same key as a configured label, the value will be overwritten.

  • obsolete secrets created by vault-kubernetes-synchronizer will be deleted

Secret Mapping

MappingVaultKubernetes
secret/k8s/firstsecret/k8s/firstfirst
secret/k8s/first:thirdsecret/k8s/firstthird
-----------------------------------------------------------------
secret/k8s/secret/k8s/firstfirst
secret/k8s/secondsecond

labels/names in Kubernetes will be validated according to RFC-1123

Encoding

If you have to encode the secret to put it in Vault, e.g. a Java KeyStore (JKS), then you can use base64 and add the prefix "base64:" to the secret in Vault

Create the secret for Vault as follows:

echo "base64:$(base64 -w0 filename)"

vault-kubernetes-synchronizer will decode the secret from Vault before creating a Kubernetes secret, to prevent double encoding.

Configuration

  • VAULT_TOKEN_PATH - the destination path on disk to store the token. Usually this is a shared volume.

  • VAULT_SECRETS - comma separated list of secrets (see Secret Mapping)

  • SECRET_PREFIX - prefix for synchronized secrets (e.g. for SECRET_PREFIX="v3t_" Vault secret "first" will get secret "v3t_first" in k8s)

  • SYNCHRONIZER_ANNOTATION - annotation used to track managed secrets (default value vault-secret). Can be very usefull if you need more than one vault-synchronizer init container in the same namespace.

  • SYNCHRONIZER_LABELS - labels will be added to every synchronized secret. Multiple key-value pairs can be separated with a comma. For each key-value pair a key and the equal sign are mandatory. Example: "k1=v1,k2=v2,k3=,k4" k4 will be ignored because the equal sign is missing.

set ALLOW_FAIL="true" for vault-kubernetes-authenticator

Error handling

If Vault authentication fails in vault-kubernetes-authenticator and ALLOW_FAIL="true" has been set for vault-kubernetes-authenticator the failed authentication will be handeled as follows:

  • all secrets in VAULT_SECRETS are available in the namespace (the content of the secrets will not be considered)- vault-kubernetes-synchronizer issues a warning and terminates successfullly.
  • any secret from VAULT_SECRETS is missing in the namespace vault-secret-synchronizer fails.

Example

Two secrets in Vault:

$ vault kv get secret/k8s/first
====== Metadata ======
...
=== Data ===
Key    Value
---    -----
one    12345678
two    23456781
$ vault kv get secret/k8s/second
====== Metadata ======
...
===== Data =====
Key       Value
---       -----
green     lantern
poison    ivy

Configure the two secrets for synchronisation with the environment variable VAULT_SECRETS:

$ vi deployment.yaml
...
    - name: VAULT_SECRETS
      value: secret/data/k8s/first,secret/data/k8s/second
...
$ k logs vault-kubernetes-synchronizer-6875c88858-t6hdw -c vault-kubernetes-authenticator
2018/11/26 14:56:30 successfully authenticated to vault
2018/11/26 14:56:30 successfully stored vault token at /home/vault/.vault-token

$ k logs vault-kubernetes-synchronizer-6875c88858-t6hdw -c vault-kubernetes-synchronizer
2018/11/26 14:56:31 read secret/data/k8s-np/appl-vault-dev-e1/first from vault
2018/11/26 14:56:31 create secret third from vault secret secret/data/k8s-np/appl-vault-dev-e1/first
2018/11/26 14:56:31 read secret/data/k8s-np/appl-vault-dev-e1/first from vault
2018/11/26 14:56:31 create secret first from vault secret secret/data/k8s-np/appl-vault-dev-e1/first
2018/11/26 14:56:31 read secret/data/k8s-np/appl-vault-dev-e1/second from vault
2018/11/26 14:56:31 create secret second from vault secret secret/data/k8s-np/appl-vault-dev-e1/second
2018/11/26 14:56:31 secrets successfully synchronized

$ k get secrets | grep -e first -e second -e third
first                                Opaque                                2      16m
second                               Opaque                                2      16m
third                                Opaque                                2      16m

$ k describe secrets first second third
Name:         first
Namespace:    vault-test
Labels:       <none>
Annotations:  vault-secret=secret/data/k8s/first

Type:  Opaque

Data
====
one:  8 bytes
two:  8 bytes


Name:         second
Namespace:    vault-test
Labels:       <none>
Annotations:  vault-secret=secret/data/k8s/second

Type:  Opaque

Data
====
poison:  3 bytes
green:   7 bytes


Name:         third
Namespace:    vault-test
Labels:       <none>
Annotations:  vault-secret=secret/data/k8s/first

Type:  Opaque

Data
====
one:  8 bytes
two:  8 bytes

Example - with failed authentication

ALLOW_FAIL="false" set for vault-kubernetes-authenticator

$ k logs vault-kubernetes-synchronizer-6875c88858-mbdsp -c vault-kubernetes-authenticator
2018/11/26 15:26:01 authentication failed: login failed with role from environment variable VAULT_ROLE: "k8s-np-appl-vault-dev-e1-auth": Put http://vault-dev-server.appl-vault-dev-e1.svc.cluster.local:8200/v1/auth/k8s-np/login: dial tcp 10.127.21.136:8200: i/o timeout

$ k logs vault-kubernetes-synchronizer-6875c88858-mbdsp -c vault-kubernetes-synchronizer
Error from server (BadRequest): container "vault-kubernetes-synchronizer" in pod "vault-kubernetes-synchronizer-6875c88858-mbdsp" is waiting to start: PodInitializing

$ k get pods
NAME                                             READY   STATUS                  RESTARTS   AGE
vault-kubernetes-synchronizer-6875c88858-mbdsp   0/1     Init:CrashLoopBackOff   3          7m40s

ALLOW_FAIL="true" set for vault-kubernetes-authenticator

$ k logs vault-kubernetes-synchronizer-7d5f65895-2pf4j -c vault-kubernetes-authenticator -f
2018/11/26 15:36:53 authentication failed - ALLOW_FAIL is set therefore pod will continue: login failed with role from environment variable VAULT_ROLE: "k8s-np-appl-vault-dev-e1-auth": Put http://vault-dev-server.appl-vault-dev-e1.svc.cluster.local:8200/v1/auth/k8s-np/login: dial tcp 10.127.21.136:8200: i/o timeout

$ k logs vault-kubernetes-synchronizer-7d5f65895-2pf4j -c vault-kubernetes-synchronizer
2018/11/26 15:36:55 check secret second from vault secret secret/data/k8s-np/appl-vault-dev-e1/second
2018/11/26 15:36:55 check secret third from vault secret secret/data/k8s-np/appl-vault-dev-e1/first
2018/11/26 15:36:55 check secret first from vault secret secret/data/k8s-np/appl-vault-dev-e1/first
2018/11/26 15:36:55 cannot synchronize secrets - all secrets seems to be available therefore pod creation will continue: could not get vault token: open /home/vault/.vault-token: no such file or directory

$ k get pods
NAME                                            READY   STATUS    RESTARTS   AGE
vault-kubernetes-synchronizer-7d5f65895-2pf4j   1/1     Running   0          5m18s

Example - with failed synchronizer

Permission issue

$ k logs pod/vault-kubernetes-synchronizer-demo-vvzxr -c vault-kubernetes-synchronizer
2019/08/20 14:00:28 Using annotation [ vault-secret ] to detect managed secrets
2019/08/20 14:00:28 failed to prepare synchronization of secrets: Error making API request.

URL: GET http://example.com/v1/sys/mounts
Code: 403. Errors:

* 1 error occurred:
        * permission denied

The fix for this is to add read permission to the read permission in the sys/mounts for the SA.

path "sys/mounts" {
 capabilities = ["read"]
}

KV/Vault engine Version missing

$ k logs pod/vault-kubernetes-synchronizer-sd-67fb88c95b-d7pkb -c vault-kubernetes-authenticator
2019/09/06 08:58:55 successfully authenticated to vault
2019/09/06 08:58:55 successfully stored vault token at /home/vault/.vault-token
$ k logs pod/vault-kubernetes-synchronizer-sd-67fb88c95b-d7pkb -c vault-kubernetes-synchronizer
2019/09/06 09:00:40 Using annotation [ vault-secret ] to detect managed secrets
2019/09/06 09:00:40 failed to prepare synchronization of secrets: strconv.Atoi: parsing "": invalid syntax

The reason for the above error is no versioning enabled for the kv secret engine. The version(1/2) has to be enabled & leaving it blank will cause above issue. Please follow the steps mentioned to fix it.

$ vault secrets list -detailed
Path                Plugin       Accessor              Default TTL    Max TTL    Force No Cache    Replication    Seal Wrap    Options    Description                                                UUID
----                ------       --------              -----------    -------    --------------    -----------    ---------    -------    -----------                                                ----
secret/             kv           kv_8210532d           system         system     false             replicated     false        map[]      n/a                                                        1dd5df15-8178-7843-6795-f05def3c3db8
$ vault secrets enable -version=1 kv
Success! Enabled the kv secrets engine at: kv/
$ vault kv enable-versioning secret/
Success! Tuned the secrets engine at: secret/
$ vault secrets list -detailed | grep kv
kv/                 kv           kv_894f5894           system         system     false             replicated     false        map[version:1]    n/a                                                        f0736f4d-343d-e32a-b2c5-897bf3552f1f
secret/             kv           kv_8210532d           system         system     false             replicated     false        map[version:2]    n/a                                                        1dd5df15-8178-7843-6795-f05def3c3db8

Example - Using labels

Initial synchronized secrets: $ k get secrets | grep ^vault- | grep -v token vault-alpha Opaque 1 26m vault-beta Opaque 1 26m vault-first Opaque 2 26m vault-gamma Opaque 1 26m vault-second Opaque 2 26m vault-third Opaque 2 26m

Add labels for some secrets:

$ for i in alpha beta gamma; do printf "labels of secret %12s: %s\n" vault-$i $(k get secret vault-${i} -o=jsonpath="{.metadata['labels']}"); done
labels of secret  vault-alpha: {"batman":"unkown","jocker":"jack_napier","superman":"unknown"}
labels of secret   vault-beta: {"batman":"bruce_wayne","joker":"jack_napier"}
labels of secret  vault-gamma: {"superman":"kal-el"}

Add SYNCHRONIZER_LABELS to your deployment:

$ vi deployment.yaml
...
        - name: SYNCHRONIZER_LABELS
          value: batman=bruce_wayne,superman=kal-el
...
> All synchronized secrets will get these labels.

Redeploy and check the labels:

$ for i in alpha beta gamma; do printf "labels of secret %12s: %s\n" vault-$i $(k get secret vault-${i} -o=jsonpath="{.metadata['labels']}"); done labels of secret vault-alpha: {"batman":"bruce_wayne","jocker":"jack_napier","superman":"kal-el"} labels of secret vault-beta: {"batman":"bruce_wayne","joker":"jack_napier","superman":"kal-el"} labels of secret vault-gamma: {"batman":"bruce_wayne","superman":"kal-el"}


> Existing labels are retained or overwritten.

## Example - Custom annotation

Set our custom annotation:

$ vi deployment.yaml ... - name: SYNCHRONIZER_ANNOTATION value: synchronized ...


Deploy and check the annotations:

$ for i in alpha beta gamma; do printf "annotations of secret %12s: %s\n" vault-$i $(k get secret vault-${i} -o=jsonpath="{.metadata['annotations']}"); done annotations of secret vault-alpha: {"synchronized":"secret/e1-k8s-pfnet-a/scratch-sauterm/greek/alpha"} annotations of secret vault-beta: {"synchronized":"secret/e1-k8s-pfnet-a/scratch-sauterm/greek/beta"} annotations of secret vault-gamma: {"synchronized":"secret/e1-k8s-pfnet-a/scratch-sauterm/greek/gamma"}


Change your custom annotation:

$ vi deployment.yaml ... - name: SYNCHRONIZER_ANNOTATION value: vault-kubernetes-synchronizer ...


Deploy and check the logs of your vault-kubernetes-synchronizer pod:

2021/01/25 11:19:33 read secret/e1-k8s-pfnet-a/scratch-sauterm/greek/alpha from vault 2021/01/25 11:19:33 WARNING: ignoring secret vault-alpha - not managed by synchronizer 2021/01/25 11:19:33 read secret/e1-k8s-pfnet-a/scratch-sauterm/greek/beta from vault 2021/01/25 11:19:33 WARNING: ignoring secret vault-beta - not managed by synchronizer 2021/01/25 11:19:33 read secret/e1-k8s-pfnet-a/scratch-sauterm/greek/gamma from vault 2021/01/25 11:19:33 WARNING: ignoring secret vault-gamma - not managed by synchronizer 2021/01/25 11:19:33 read secret/e1-k8s-pfnet-a/scratch-sauterm/first from vault 2021/01/25 11:19:33 WARNING: ignoring secret vault-first - not managed by synchronizer 2021/01/25 11:19:33 read secret/e1-k8s-pfnet-a/scratch-sauterm/second from vault 2021/01/25 11:19:33 WARNING: ignoring secret vault-second - not managed by synchronizer 2021/01/25 11:19:33 read secret/e1-k8s-pfnet-a/scratch-sauterm/first from vault 2021/01/25 11:19:33 WARNING: ignoring secret vault-third - not managed by synchronizer


> Changing the annotation does not work. You have to delete the secrets first.

# Sidecar _vault-kubernetes-token-renewer_

Depends on Init Container _vault-kubernetes-authenticator_

- renew the Vault token regularly

## Configuration

- VAULT_TOKEN_PATH - the destination path on disk to store the token. Usually this is a shared volume.
- VAULT_REAUTH - re-authenticate if the token is invalid (default: "false")
- VAULT_TTL - requested token ttl (can be overwritten by Vault)

> If you set VAULT_REAUTH to "true", you have to provide all necessary environment variable for authentication (see: _vault-kubernetes-authenticator_). The token changes when re-authentication happens and must therefore be read again.

## Example

$ k logs vault-kubernetes-token-renewer-844488f7bc-c6ztf -c vault-kubernetes-authenticator 2018/11/26 14:56:30 successfully authenticated to vault 2018/11/26 14:56:30 successfully stored vault token at /home/vault/.vault-token

$ k logs vault-kubernetes-token-renewer-844488f7bc-c6ztf -c vault-kubernetes-token-renewer 2018/11/26 14:56:32 start renewer loop 2018/11/26 14:56:32 token renewed



# Build

Install [mage](https://magefile.org/)

> The `DOCKER_TARGET` environment variable will be used to tag and push the images. If not set, the images will not be tagged and pushed.

$ export GO111MODULE=on $ export DOCKER_TARGET="registry.example.com/repopath" $ mage buildAllImages



# Demo

- Edit `profile`

cd demo ./deploy.sh profile ... ./delete.sh namespace



# Links

- [Using HashiCorp Vault with Kubernetes (Cloud Next '18)](https://www.youtube.com/watch?v=B16YTeSs1hI)
- [Github - vault-kubernetes-authenticator](https://github.com/sethvargo/vault-kubernetes-authenticator)
- [Vault - Kubernetes Auth Method](https://www.vaultproject.io/docs/auth/kubernetes.html)
- [Kubernetes - Init Containers](https://kubernetes.io/docs/concepts/workloads/pods/init-containers)