HashiCorp Vault
Hashicorp Vault
External Secrets Operator integrates with HashiCorp Vault for secret management.
The KV Secrets Engine is the only one supported by this provider. For other secrets engines, please refer to the Vault Generator.
Example
First, create a SecretStore with a vault backend. For the sake of simplicity we'll use a static token root
:
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend
spec:
provider:
vault:
server: "http://my.vault.server:8200"
path: "secret"
# Version is the Vault KV secret engine version.
# This can be either "v1" or "v2", defaults to "v2"
version: "v2"
auth:
# points to a secret that contains a vault token
# https://www.vaultproject.io/docs/auth/token
tokenSecretRef:
name: "vault-token"
key: "token"
---
apiVersion: v1
kind: Secret
metadata:
name: vault-token
data:
token: cm9vdA== # "root"
ClusterSecretStore
, Be sure to provide namespace
for tokenSecretRef
with the namespace of the secret that we just created.
Then create a simple k/v pair at path secret/foo
:
vault kv put secret/foo my-value=s3cr3t
Can check kv version using following and check for Options
column, it should indicate [version:2]:
vault secrets list -detailed
If you are using version: 1, just remember to update your SecretStore manifest appropriately
Now create a ExternalSecret that uses the above SecretStore:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: vault-example
spec:
refreshInterval: "15s"
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: example-sync
data:
- secretKey: foobar
remoteRef:
key: foo
property: my-value
# metadataPolicy to fetch all the labels in JSON format
- secretKey: tags
remoteRef:
metadataPolicy: Fetch
key: foo
# metadataPolicy to fetch a specific label (dev) from the source secret
- secretKey: developer
remoteRef:
metadataPolicy: Fetch
key: foo
property: dev
---
# will create a secret with:
kind: Secret
metadata:
name: example-sync
data:
foobar: czNjcjN0
Keep in mind that fetching the labels with metadataPolicy: Fetch
only works with KV sercrets engine version v2.
Fetching Raw Values
You can fetch all key/value pairs for a given path If you leave the remoteRef.property
empty. This returns the json-encoded secret value for that path.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: vault-example
spec:
# ...
data:
- secretKey: foobar
remoteRef:
key: /dev/package.json
Nested Values
Vault supports nested key/value pairs. You can specify a gjson expression at remoteRef.property
to get a nested value.
Given the following secret - assume its path is /dev/config
:
{
"foo": {
"nested": {
"bar": "mysecret"
}
}
}
You can set the remoteRef.property
to point to the nested key using a gjson expression.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: vault-example
spec:
# ...
data:
- secretKey: foobar
remoteRef:
key: /dev/config
property: foo.nested.bar
---
# creates a secret with:
# foobar=mysecret
If you would set the remoteRef.property
to just foo
then you would get the json-encoded value of that property: {"nested":{"bar":"mysecret"}}
.
Multiple nested Values
You can extract multiple keys from a nested secret using dataFrom
.
Given the following secret - assume its path is /dev/config
:
{
"foo": {
"nested": {
"bar": "mysecret",
"baz": "bang"
}
}
}
You can set the remoteRef.property
to point to the nested key using a gjson expression.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: vault-example
spec:
# ...
dataFrom:
- extract:
key: /dev/config
property: foo.nested
That results in a secret with these values:
bar=mysecret
baz=bang
Getting multiple secrets
You can extract multiple secrets from Hashicorp vault by using dataFrom.Find
Currently, dataFrom.Find
allows users to fetch secret names that match a given regexp pattern, or fetch secrets whose custom_metadata
tags match a predefined set.
Warning
The way hashicorp Vault currently allows LIST operations is through the existence of a secret metadata. If you delete the secret, you will also need to delete the secret's metadata or this will currently make Find operations fail.
Given the following secret - assume its path is /dev/config
:
{
"foo": {
"nested": {
"bar": "mysecret",
"baz": "bang"
}
}
}
Also consider the following secret has the following custom_metadata
:
{
"environment": "dev",
"component": "app-1"
}
It is possible to find this secret by all the following possibilities:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: vault-example
spec:
# ...
dataFrom:
- find: #will return every secret with 'dev' in it (including paths)
name:
regexp: dev
- find: #will return every secret matching environment:dev tags from dev/ folder and beyond
tags:
environment: dev
{
"dev_config":"{\"foo\":{\"nested\":{\"bar\":\"mysecret\",\"baz\":\"bang\"}}}"
}
Currently, Find
operations are recursive throughout a given vault folder, starting on provider.Path
definition. It is recommended to narrow down the scope of search by setting a find.path
variable. This is also useful to automatically reduce the resulting secret key names:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: vault-example
spec:
# ...
dataFrom:
- find: #will return every secret from dev/ folder
path: dev
name:
regexp: ".*"
- find: #will return every secret matching environment:dev tags from dev/ folder
path: dev
tags:
environment: dev
{
"config":"{\"foo\": {\"nested\": {\"bar\": \"mysecret\",\"baz\": \"bang\"}}}"
}
Authentication
We support five different modes for authentication: token-based, appRole, kubernetes-native, ldap, jwt/oidc and awsAuth, each one comes with it's own trade-offs. Depending on the authentication method you need to adapt your environment.
Token-based authentication
A static token is stored in a Kind=Secret
and is used to authenticate with vault.
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend
namespace: example
spec:
provider:
vault:
server: "https://vault.acme.org"
path: "secret"
version: "v2"
auth:
# points to a secret that contains a vault token
# https://www.vaultproject.io/docs/auth/token
tokenSecretRef:
name: "my-secret"
key: "vault-token"
ClusterSecretStore
, Be sure to provide namespace
in tokenSecretRef
with the namespace where the secret resides.
AppRole authentication example
AppRole authentication reads the secret id from a
Kind=Secret
and uses the specified roleId
to aquire a temporary token to fetch secrets.
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend
namespace: example
spec:
provider:
vault:
server: "https://vault.acme.org"
path: "secret"
version: "v2"
auth:
# VaultAppRole authenticates with Vault using the
# App Role auth mechanism
# https://www.vaultproject.io/docs/auth/approle
appRole:
# Path where the App Role authentication backend is mounted
path: "approle"
# RoleID configured in the App Role authentication backend
roleId: "db02de05-fa39-4855-059b-67221c5c2f63"
# Reference to a key in a K8 Secret that contains the App Role SecretId
secretRef:
name: "my-secret"
key: "secret-id"
ClusterSecretStore
, Be sure to provide namespace
in secretRef
with the namespace where the secret resides.
Kubernetes authentication
Kubernetes-native authentication has three options of obtaining credentials for vault:
- by using a service account jwt referenced in
serviceAccountRef
- by using the jwt from a
Kind=Secret
referenced by thesecretRef
- by using transient credentials from the mounted service account token within the external-secrets operator
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend
namespace: example
spec:
provider:
vault:
server: "https://vault.acme.org"
path: "secret"
version: "v2"
auth:
# Authenticate against Vault using a Kubernetes ServiceAccount
# token stored in a Secret.
# https://www.vaultproject.io/docs/auth/kubernetes
kubernetes:
# Path where the Kubernetes authentication backend is mounted in Vault
mountPath: "kubernetes"
# A required field containing the Vault Role to assume.
role: "demo"
# Optional service account field containing the name
# of a kubernetes ServiceAccount
serviceAccountRef:
name: "my-sa"
# Optional secret field containing a Kubernetes ServiceAccount JWT
# used for authenticating with Vault
secretRef:
name: "my-secret"
key: "vault"
ClusterSecretStore
, Be sure to provide namespace
in serviceAccountRef
or in secretRef
, if used.
LDAP authentication
LDAP authentication uses
username/password pair to get an access token. Username is stored directly in
a Kind=SecretStore
or Kind=ClusterSecretStore
resource, password is stored
in a Kind=Secret
referenced by the secretRef
.
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend
namespace: example
spec:
provider:
vault:
server: "https://vault.acme.org"
path: "secret"
version: "v2"
auth:
# VaultLdap authenticates with Vault using the LDAP auth mechanism
# https://www.vaultproject.io/docs/auth/ldap
ldap:
# Path where the LDAP authentication backend is mounted
path: "ldap"
# LDAP username
username: "username"
secretRef:
name: "my-secret"
key: "ldap-password"
ClusterSecretStore
, Be sure to provide namespace
in secretRef
with the namespace where the secret resides.
JWT/OIDC authentication
JWT/OIDC uses either a
JWT token stored in a Kind=Secret
and referenced by the
secretRef
or a temporary Kubernetes service account token retrieved via the TokenRequest
API. Optionally a role
field can be defined in a Kind=SecretStore
or Kind=ClusterSecretStore
resource.
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend
namespace: example
spec:
provider:
vault:
server: "https://vault.acme.org"
path: "secret"
version: "v2"
auth:
# VaultJwt authenticates with Vault using the JWT/OIDC auth mechanism
# https://www.vaultproject.io/docs/auth/jwt
jwt:
# Path where the JWT authentication backend is mounted
path: "jwt"
# JWT role configured in a Vault server, optional.
role: "vault-jwt-role"
# Retrieve JWT token from a Kubernetes secret
secretRef:
name: "my-secret"
key: "jwt-token"
# ... or retrieve a Kubernetes service account token via the `TokenRequest` API
kubernetesServiceAccountToken:
serviceAccountRef:
name: "my-sa"
# `audiences` defaults to `["vault"]` it not supplied
audiences:
- vault
# `expirationSeconds` defaults to 10 minutes if not supplied
expirationSeconds: 600
ClusterSecretStore
, Be sure to provide namespace
in secretRef
with the namespace where the secret resides.
AWS IAM authentication
AWS IAM uses either a
set of AWS Programmatic access credentials stored in a Kind=Secret
and referenced by the
secretRef
or by getting the authentication token from an IRSA enabled service account
Access Key ID & Secret Access Key
You can store Access Key ID & Secret Access Key in a Kind=Secret
and reference it from a SecretStore.
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend-aws-iam
spec:
provider:
vault:
server: "http://my.vault.server:8200"
path: secret
version: v2
namespace: <vault_namespace>
auth:
iam:
# Path where the AWS auth method is enabled in Vault, e.g: "aws/". Defaults to aws
path: aws
# AWS Region. Defaults to us-east-1
region: us-east-1
# optional: assume role before fetching secrets
role: arn:aws:iam::1234567890:role/role-a
# Vault Role. In vault, a role describes an identity with a set of permissions, groups, or policies you want to attach a user of the secrets engine
vaultRole: vault-role-for-aws-iam-auth
# Optional. Placeholder to supply header X-Vault-AWS-IAM-Server-ID. It is an additional (optional) header used by Vault IAM auth method to mitigate against different types of replay attacks. More details here: https://developer.hashicorp.com/vault/docs/auth/aws
vaultAwsIamServerID: example-vaultAwsIamServerID
secretRef: #Use this method when you have static AWS creds.
accessKeyIDSecretRef:
name: vault-iam-creds-secret
key: access-key
secretAccessKeySecretRef:
name: vault-iam-creds-secret
key: secret-access-key
sessionTokenSecretRef:
name: vault-iam-creds-secret
key: secret-session-token
NOTE: In case of a ClusterSecretStore
, Be sure to provide namespace
in accessKeyIDSecretRef
, secretAccessKeySecretRef
with the namespaces where the secrets reside.
EKS Service Account credentials
This feature lets you use short-lived service account tokens to authenticate with AWS. You must have Service Account Volume Projection enabled - it is by default on EKS. See EKS guide on how to set up IAM roles for service accounts.
The big advantage of this approach is that ESO runs without any credentials.
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/my-irsa-enabled-role
name: my-serviceaccount
namespace: default
Reference the service account from above in the Secret Store:
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend-aws-iam
spec:
provider:
vault:
server: "http://my.vault.server:8200"
path: secret
version: v2
namespace: <vault_namespace>
auth:
iam:
# Path where the AWS auth method is enabled in Vault, e.g: "aws/". Defaults to aws
path: aws
# AWS Region. Defaults to us-east-1
region: us-east-1
# Vault Role. In vault, a role describes an identity with a set of permissions, groups, or policies you want to attach a user of the secrets engine
vaultRole: vault-role-for-aws-iam-auth
# Optional. Placeholder to supply header X-Vault-AWS-IAM-Server-ID. It is an additional (optional) header used by Vault IAM auth method to mitigate against different types of replay attacks. More details here: https://developer.hashicorp.com/vault/docs/auth/aws
vaultAwsIamServerID: example-vaultAwsIamServerID
jwt:
serviceAccountRef:
name: my-serviceaccount #Provide service account with IRSA enabled
Controller's Pod Identity
This is basicially a zero-configuration authentication approach that inherits the credentials from the controller's pod identity
This approach assumes that appropriate IRSA setup is done controller's pod (i.e. IRSA enabled IAM role is created appropriately and controller's service account is annotated appropriately with the annotation "eks.amazonaws.com/role-arn" to enable IRSA)
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend-aws-iam
spec:
provider:
vault:
server: "http://my.vault.server:8200"
path: secret
version: v2
namespace: <vault_namespace>
auth:
iam:
# Path where the AWS auth method is enabled in Vault, e.g: "aws/". Defaults to aws
path: aws
# AWS Region. Defaults to us-east-1
region: us-east-1
# Vault Role. In vault, a role describes an identity with a set of permissions, groups, or policies you want to attach a user of the secrets engine
vaultRole: vault-role-for-aws-iam-auth
# Optional. Placeholder to supply header X-Vault-AWS-IAM-Server-ID. It is an additional (optional) header used by Vault IAM auth method to mitigate against different types of replay attacks. More details here: https://developer.hashicorp.com/vault/docs/auth/aws
vaultAwsIamServerID: example-vaultAwsIamServerID
NOTE: In case of a ClusterSecretStore
, Be sure to provide namespace
for serviceAccountRef
with the namespace where the service account resides.
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend
namespace: example
spec:
provider:
vault:
server: "https://vault.acme.org"
path: "secret"
version: "v2"
auth:
# VaultJwt authenticates with Vault using the JWT/OIDC auth mechanism
# https://www.vaultproject.io/docs/auth/jwt
jwt:
# Path where the JWT authentication backend is mounted
path: "jwt"
# JWT role configured in a Vault server, optional.
role: "vault-jwt-role"
# Retrieve JWT token from a Kubernetes secret
secretRef:
name: "my-secret"
key: "jwt-token"
# ... or retrieve a Kubernetes service account token via the `TokenRequest` API
kubernetesServiceAccountToken:
serviceAccountRef:
name: "my-sa"
# `audiences` defaults to `["vault"]` it not supplied
audiences:
- vault
# `expirationSeconds` defaults to 10 minutes if not supplied
expirationSeconds: 600
ClusterSecretStore
, Be sure to provide namespace
in secretRef
with the namespace where the secret resides.
PushSecret
Vault supports PushSecret features which allow you to sync a given kubernetes secret key into a hashicorp vault secret. In order to do so, it is expected that the secret key is a valid JSON object.
In order to use PushSecret, you need to give create
, read
and update
permissions to the path where you want to push secrets to for both data
and metadata
of the secret. Use it with care!
Here is an example on how to set it up:
apiVersion: v1
kind: Secret
metadata:
name: source-secret
namespace: default
stringData:
source-key: "{\"foo\":\"bar\"}" # Needs to be a JSON
---
apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
name: pushsecret-example
namespace: default
spec:
refreshInterval: 10s # Refresh interval for which push secret will reconcile
secretStoreRefs: # A list of secret stores to push secrets to
- name: vault-secretstore
kind: SecretStore
selector:
secret:
name: source-secret # Source Kubernetes secret to be pushed
data:
- match:
secretKey: source-key # Source Kubernetes secret key containing the vault secret (in JSON format)
remoteRef:
remoteKey: vault/secret # path to vault secret. This path is appended with the vault-store path.
Vault Enterprise
Eventual Consistency and Performance Standby Nodes
When using Vault Enterprise with performance standby nodes, any follower can handle read requests immediately after the provider has authenticated. Since Vault becomes eventually consistent in this mode, these requests can fail if the login has not yet propagated to each server's local state.
Below are two different solutions to this scenario. You'll need to review them and pick the best fit for your environment and Vault configuration.
Vault Namespaces
Vault namespaces are an enterprise feature that support multi-tenancy. You can specify a vault namespace using the namespace
property when you define a SecretStore:
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend
spec:
provider:
vault:
server: "http://my.vault.server:8200"
# See https://www.vaultproject.io/docs/enterprise/namespaces
namespace: "ns1"
path: "secret"
version: "v2"
auth:
# ...
Read Your Writes
Vault 1.10.0 and later encodes information in the token to detect the case when a server is behind. If a Vault server does not have information about the provided token, Vault returns a 412 error so clients know to retry.
A method supported in versions Vault 1.7 and later is to utilize the
X-Vault-Index
header returned on all write requests (including logins).
Passing this header back on subsequent requests instructs the Vault client
to retry the request until the server has an index greater than or equal
to that returned with the last write. Obviously though, this has a performance
hit because the read is blocked until the follower's local state has caught up.
Forward Inconsistent
Vault also supports proxying inconsistent requests to the current cluster leader for immediate read-after-write consistency.
Vault 1.10.0 and later support a replication configuration that detects when forwarding should occur and does it transparently to the client.
In Vault 1.7 forwarding can be achieved by setting the X-Vault-Inconsistent
header to forward-active-node
. By default, this behavior is disabled and must
be explicitly enabled in the server's replication configuration.