CyberArk Secrets Manager Provider
This section describes how to set up the CyberArk Secrets Manager provider for External Secrets Operator (ESO). For a working example, see the Accelerator-K8s-External-Secrets repo.
Prerequisites
Before installing the Secrets Manager provider, you need:
- A running instance of Conjur OSS or CyberArk Secrets Manager, with:
- An accessible Secrets Manager endpoint (for example:
https://myapi.example.com). - Your configured Secrets Manager authentication info (such as
hostid,apikeyor service ID of JWT or Cert authenticator). For more information on configuring Secrets Manager, see Policy statement reference. - Support for your authentication method (
apikeyis supported by default,jwtandcertrequire additional configuration). - Optional: Secrets Manager server certificate (see below).
- A Kubernetes cluster with ESO installed.
Secrets Manager server certificate
If you set up your Secrets Manager server with a self-signed certificate, we recommend that you populate the caBundle field with the Secrets Manager self-signed certificate in the secret-store definition. The certificate CA must be referenced in the secret-store definition using either caBundle or caProvider:
....
spec:
provider:
conjur:
# Service URL
url: https://myapi.conjur.org
# [OPTIONAL] base64 encoded string of certificate
caBundle: "<base64 encoded cabundle>"
# [OPTIONAL] caProvider:
# Instead of caBundle you can also specify a caProvider,
# which retrieves the cert from a Secret or ConfigMap
caProvider:
type: "Secret" # Can be Secret or ConfigMap
name: "<name of secret or configmap>"
key: "<key inside secret or configmap>"
# namespace is required for ClusterSecretStore
# but not relevant for SecretStore
namespace: "my-cert-secret-namespace"
....
External secret store
To synchronise secrets with Secrets Manager, you must configure an external secret store with one of the following three methods to authenticate to the Secrets Manager API:
apikey: uses a Secrets Managerhostidandapikeyto authenticate with Secrets Managerjwt: uses a JWT to authenticate with Secrets Managercert: uses a client certificate and private key to authenticate with Secrets Manager
Option 1: External secret store with apiKey authentication
This method uses a Secrets Manager hostid and apikey to authenticate with Secrets Manager. It is the simplest method to set up and use because your Secrets Manager instance requires no additional configuration.
Step 1: Define an external secret store
Tip
Save as the file as: conjur-secret-store.yaml
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: conjur
spec:
provider:
conjur:
# Service URL
url: https://myapi.conjur.org
# [OPTIONAL] base64 encoded string of certificate
caBundle: OPTIONALxFIELDxxxBase64xCertxString==
auth:
apikey:
# conjur account
account: conjur
userRef: # Get this from K8S secret
name: conjur-creds
key: hostid
apiKeyRef: # Get this from K8S secret
name: conjur-creds
key: apikey
Step 2: Create Kubernetes secrets for Secrets Manager credentials
To connect to the Secrets Manager server, the ESO Secrets Manager provider needs to retrieve the apikey credentials from K8s secrets.
Note
For more information about how to create K8s secrets, see Creating a secret.
Here is an example of how to create K8s secrets using the kubectl command:
# This is all one line
kubectl -n external-secrets create secret generic conjur-creds --from-literal=hostid=MYCONJURHOSTID --from-literal=apikey=MYAPIKEY
# Example:
# kubectl -n external-secrets create secret generic conjur-creds --from-literal=hostid=host/data/app1/host001 --from-literal=apikey=321blahblah
Note
conjur-creds is the name defined in the userRef and apikeyRef fields of the conjur-secret-store.yml file.
Step 3: Create the external secrets store
Important
Unless you are using a ClusterSecretStore, credentials must reside in the same namespace as the SecretStore.
# WARNING: creates the store in the "external-secrets" namespace, update the value as needed
#
kubectl apply -n external-secrets -f conjur-secret-store.yaml
# WARNING: running the delete command will delete the secret store configuration
#
# If there is a need to delete the external secretstore
# kubectl delete secretstore -n external-secrets conjur
Option 2: External secret store with JWT authentication
This method uses JWT tokens to authenticate with Secrets Manager. You can use the following methods to retrieve a JWT token for authentication:
- JWT token from a referenced Kubernetes service account
- JWT token stored in a Kubernetes secret
Step 1: Define an external secret store
When you use JWT authentication, the following must be specified in the SecretStore:
account- The name of the Secrets Manager accountserviceId- The ID of the JWT AuthenticatorWebServiceconfigured in Secrets Manager that is used to authenticate the JWT token
You can retrieve the JWT token from either a referenced service account or a Kubernetes secret.
For example, to retrieve a JWT token from a referenced Kubernetes service account, the following secret store definition can be used:
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: conjur
spec:
provider:
conjur:
# Service URL
url: https://myapi.conjur.org
# [OPTIONAL] base64 encoded string of certificate
caBundle: OPTIONALxFIELDxxxBase64xCertxString==
auth:
jwt:
# conjur account
account: conjur
# The authn-jwt service ID
serviceID: my-jwt-auth-service
# Service account to retrieve JWT token for
serviceAccountRef:
name: my-service-account
# [OPTIONAL] audiences to include in JWT token
audiences:
- https://conjur.company.com
Important
This method is only supported in Kubernetes 1.22 and above as it uses the TokenRequest API to get the JWT token from the referenced service account. Audiences can be defined in the Secrets Manager JWT authenticator.
Alternatively, here is an example where a secret containing a valid JWT token is referenced:
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: conjur
spec:
provider:
conjur:
# Service URL
url: https://myapi.conjur.org
# [OPTIONAL] base64 encoded string of certificate
caBundle: OPTIONALxFIELDxxxBase64xCertxString==
auth:
jwt:
# conjur account
account: conjur
# The authn-jwt service ID
serviceID: my-jwt-auth-service
# Secret containing a valid JWT token
secretRef:
name: my-jwt-secret
key: token
The JWT token must identify your Secrets Manager host, be compatible with your configured Secrets Manager JWT authenticator, and meet all the Secrets Manager JWT guidelines.
You can use an external JWT issuer or the Kubernetes API server to create the token. For example, a Kubernetes service account token can be created with this command:
kubectl create token my-service-account --audience='https://conjur.company.com' --duration=3600s
Save the secret store file as conjur-secret-store.yaml.
Step 2: Create the external secrets store
# WARNING: creates the store in the "external-secrets" namespace, update the value as needed
#
kubectl apply -n external-secrets -f conjur-secret-store.yaml
# WARNING: running the delete command will delete the secret store configuration
#
# If there is a need to delete the external secretstore
# kubectl delete secretstore -n external-secrets conjur
Option 3: External secret store with cert authentication
This method uses a client X.509 certificate and private key to authenticate with Secrets Manager via the authn-cert authenticator. The certificate must be issued by a CA that is trusted by the Secrets Manager authn-cert service. Conjur validates the presented certificate using constraints configured in the authenticator policy (such as cn, san-uri, san-dns, san-ip variables), and optionally per-host annotations that scope constraints to individual workloads.
For more information on configuring the certificate authenticator, see the Secrets Manager Certificate authenticator documentation.
Step 1: Define an external secret store
When you use cert authentication, the following must be specified in the SecretStore:
account- The name of the Secrets Manager accountserviceID- The ID of theauthn-certWebServiceconfigured in Secrets Manager that is used to authenticate the client certificateclientCertRef- Reference to a Kubernetes secret containing the PEM-encoded client certificateclientKeyRef- Reference to a Kubernetes secret containing the PEM-encoded client private keyhostId(optional) - The Secrets Manager host identity to authenticate as. Required in the defaultrequestmode. Leave empty when usingspiffemode, where the host identity is derived from the SPIFFE SAN URI (spiffe://<trust-domain>/<workload-id>) in the certificate.
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: conjur
spec:
provider:
conjur:
# Service URL
url: https://myapi.conjur.org
# [OPTIONAL] base64 encoded string of certificate
caBundle: OPTIONALxFIELDxxxBase64xCertxString==
auth:
cert:
# conjur account
account: conjur
# The authn-cert service ID
serviceID: my-cert-auth-service
# [OPTIONAL] HostID for cert authentication (leave empty for 'spiffe' mode).
# hostId: vm-01
# Reference to a Kubernetes secret containing the PEM-encoded client certificate
clientCertRef:
name: conjur-client-cert
key: tls.crt
# Reference to a Kubernetes secret containing the PEM-encoded client key
clientKeyRef:
name: conjur-client-cert
key: tls.key
Step 2: Create Kubernetes secrets for the client certificate and key
To connect to the Secrets Manager server using certificate authentication, the ESO Secrets Manager provider needs to retrieve the client certificate and private key from K8s secrets.
You can use either a TLS-typed secret or a generic secret. The following example uses a generic secret:
# This is all one line
kubectl -n external-secrets create secret generic conjur-client-cert \
--from-file=tls.crt=/path/to/client.crt \
--from-file=tls.key=/path/to/client.key
Note
conjur-client-cert is the name defined in the clientCertRef and clientKeyRef fields of the conjur-secret-store.yaml file. Both the certificate and the key must be PEM-encoded.
Step 3: Create the external secrets store
Important
Unless you are using a ClusterSecretStore, credentials must reside in the same namespace as the SecretStore.
# WARNING: creates the store in the "external-secrets" namespace, update the value as needed
#
kubectl apply -n external-secrets -f conjur-secret-store.yaml
# WARNING: running the delete command will delete the secret store configuration
#
# If there is a need to delete the external secretstore
# kubectl delete secretstore -n external-secrets conjur
Option 4: External secret store with IAM authentication
This method uses the AWS IAM authn-iam authenticator to authenticate with Secrets Manager. The ESO pod presents a signed AWS GetCallerIdentity request; Conjur verifies it against the IAM role mapped to the Conjur host. No long-lived credentials are stored — ambient credentials are used automatically (IRSA, EC2 instance profile, ECS task role, environment variables, etc.).
When you use IAM authentication, the following must be specified in the SecretStore:
account— The name of the Secrets Manager account.serviceID— The ID of the IAM AuthenticatorWebServiceconfigured in Secrets Manager.hostId— The Conjur host mapped to the AWS IAM role (e.g.data/myapp/123456789012/MyRole).
Optionally, you may reference a Kubernetes Secret that contains explicit AWS credentials via the secretRef field:
accessKeyIDSecretRef— Reference to a Secret key containing the AWS Access Key ID.secretAccessKeySecretRef— Reference to a Secret key containing the AWS Secret Access Key.sessionTokenSecretRef(optional) — Reference to a Secret key containing the AWS Session Token. Required only when using temporary credentials (for example, credentials obtained viaAssumeRoleorAssumeRoleWithWebIdentity).
If secretRef is omitted, the default AWS SDK credential chain is used.
The Secrets Manager url must use https:// when using IAM authentication, since the AWS IAM authenticator sends
a signed request as a bearer credential. If you must connect over plain HTTP (for example, a service mesh sidecar
that terminates TLS locally), set insecure: true to explicitly opt out of this check.
Step 1: Define an external secret store
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: conjur-iam
namespace: default
spec:
provider:
conjur:
url: https://conjur.example.com
# caBundle: <base64-encoded PEM> # uncomment for self-signed certs
auth:
iam:
account: myorg
serviceID: prod # authn-iam webservice ID in Conjur policy
hostId: data/myapp/123456789012/MyRole # Conjur host mapped to the AWS IAM role
# secretRef: # omit to use ambient credentials (IRSA, instance profile, etc.)
# accessKeyIDSecretRef:
# name: aws-creds
# key: access-key-id
# secretAccessKeySecretRef:
# name: aws-creds
# key: secret-access-key
# sessionTokenSecretRef: # required only when using temporary credentials
# name: aws-creds
# key: session-token
# insecure: true # required to authenticate over a plain HTTP url; use only when the connection is otherwise protected
Step 2: Create the external secrets store
kubectl apply -n external-secrets -f conjur-secret-store.yaml
Option 5: External secret store with Azure authentication
This method uses the Azure authn-azure authenticator to authenticate with Secrets Manager. The ESO pod presents an Azure JWT token; Conjur verifies it against the managed identity mapped to the Conjur host.
Two sub-modes are available:
- Ambient IMDS (recommended on AKS / Azure VM): no
serviceAccountRefis set.conjur-api-gofetches the Azure JWT from the Azure Instance Metadata Service (IMDS) automatically. Use this when the pod runs on Azure infrastructure with a managed identity attached. - ServiceAccount token (
serviceAccountRef): ESO requests a Kubernetes ServiceAccount token via the TokenRequest API and forwards it as the Azure JWT. Use this for local testing or when running outside Azure.
For user-assigned managed identities you must also set clientId to the client ID of the managed identity. For system-assigned identities, omit clientId.
Step 1: Define an external secret store (ambient IMDS)
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: conjur-azure-ambient
namespace: default
spec:
provider:
conjur:
url: https://conjur.example.com
# caBundle: <base64-encoded PEM> # uncomment for self-signed certs
auth:
azure:
account: myorg
serviceID: prod # authn-azure webservice ID in Conjur policy
hostId: data/myapp/myhost # Conjur host mapped to the Azure managed identity
# clientId: <managed-identity-client-id> # required for user-assigned managed identities
Step 1 (alternative): Define an external secret store (ServiceAccount token)
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: conjur-azure-sa
namespace: default
spec:
provider:
conjur:
url: https://conjur.example.com
# caBundle: <base64-encoded PEM> # uncomment for self-signed certs
auth:
azure:
account: myorg
serviceID: prod
hostId: data/myapp/myhost
serviceAccountRef:
name: my-service-account # Kubernetes ServiceAccount in the same namespace
audiences:
- conjur # must match the Conjur authn-azure policy audience
Step 2: Create the external secrets store
kubectl apply -n external-secrets -f conjur-secret-store.yaml
Option 6: External secret store with GCP authentication
This method uses the GCP authn-gcp authenticator to authenticate with Secrets Manager. The ESO pod presents a GCP identity token; Conjur verifies it against the service account mapped to the Conjur host.
Two sub-modes are supported:
- Ambient GCP Metadata Service (recommended on GKE with Workload Identity): no
secretRefis set.conjur-api-gofetches the GCP identity token from the GCP Metadata Service automatically. Use this when the pod runs on GKE with Workload Identity or on a GCE instance. - Kubernetes Secret (
secretRef): a pre-obtained GCP identity token is read from a Kubernetes Secret. Use this for local testing or when running outside GCP.
The following must be specified in the SecretStore:
account— The Conjur organization account name.hostId— The Conjur host mapped to the GCP service account (e.g.data/myapp/myhost).
Optionally, you may set serviceID to the GCP Authenticator WebService ID configured in Conjur policy. This field is reserved for future use; Conjur's authn-gcp authenticator does not include the service ID in the authentication URL.
Step 1: Define an external secret store (ambient GCP Metadata Service)
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: conjur-gcp-ambient
namespace: default
spec:
provider:
conjur:
url: https://conjur.example.com
# caBundle: <base64-encoded PEM> # uncomment for self-signed certs
auth:
gcp:
account: myorg
serviceID: prod # reserved for future use; not used by authn-gcp today
hostId: data/myapp/myhost # Conjur host mapped to the GCP service account
Step 1 (alternative): Define an external secret store (explicit token from Kubernetes Secret)
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: conjur-gcp-secretref
namespace: default
spec:
provider:
conjur:
url: https://conjur.example.com
# caBundle: <base64-encoded PEM> # uncomment for self-signed certs
auth:
gcp:
account: myorg
serviceID: prod # reserved for future use; not used by authn-gcp today
hostId: data/myapp/myhost
secretRef:
jwt:
name: gcp-token-secret # Kubernetes Secret in the same namespace
key: token # key within the Secret holding the GCP identity token
Step 2: Create the external secrets store
kubectl apply -n external-secrets -f conjur-secret-store.yaml
Define an external secret
After you have configured the Secrets Manager provider secret store, you can fetch secrets from Secrets Manager.
Here is an example of how to fetch a single secret from Secrets Manager:
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: conjur
spec:
refreshInterval: 1h0m0s
secretStoreRef:
# This name must match the metadata.name in the `SecretStore`
name: conjur
kind: SecretStore
data:
- secretKey: secret00
remoteRef:
key: data/app1/secret00
Save the external secret file as conjur-external-secret.yaml.
Find by Name or Find by Tag
The Secrets Manager provider also supports the Find by Name and Find by Tag ESO features. This means that you can use a regular expression or tags to dynamically fetch multiple secrets from Secrets Manager.
These two filters are mutually exclusive on a single find: if both name and tags are set, only name
is used and tags are ignored (see issue #6554).
To find secrets by name:
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: conjur-find-by-name
spec:
refreshInterval: 1h0m0s
secretStoreRef:
# This name must match the metadata.name in the `SecretStore`
name: conjur
kind: SecretStore
target:
name: k8s-secret-to-be-created
dataFrom:
- find:
name:
# Match all secrets in the app1 namespace (e.g., `app1/secret00`, `app1/secret01`, etc.)
regexp: "^app1\/.+$"
To find secrets by tag, use tags instead of name:
dataFrom:
- find:
tags:
environment: "prod"
application: "app1"
If you use these features, we strongly recommend that you limit the permissions of the Secrets Manager host to only the secrets that it needs to access. This is more secure and it reduces the load on both the Secrets Manager server and ESO.
Create the external secret
# WARNING: creates the external-secret in the "external-secrets" namespace, update the value as needed
#
kubectl apply -n external-secrets -f conjur-external-secret.yaml
# WARNING: running the delete command will delete the external-secrets configuration
#
# If there is a need to delete the external secret
# kubectl delete externalsecret -n external-secrets conjur
Get the K8s secret
- Log in to your Secrets Manager server and verify that your secret exists
- Review the value of your Kubernetes secret to verify that it contains the same value as the Secrets Manager server
# WARNING: this command will reveal the stored secret in plain text
#
# Assuming the secret name is "secret00", this will show the value
kubectl get secret -n external-secrets conjur -o jsonpath="{.data.secret00}" | base64 --decode && echo
Define a push secret
The Secrets Manager provider also supports PushSecret. This means that you can push the values of a Kubernetes secret into Secrets Manager variables.
Here is an example of how to push a single key of a Kubernetes secret into Secrets Manager:
apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
name: conjur
spec:
refreshInterval: 1h0m0s
# Only the Replace update policy is supported by the Secrets Manager provider.
updatePolicy: Replace
# Deleting pushed variables is not supported, keep the default deletion policy.
deletionPolicy: None
secretStoreRefs:
# This name must match the metadata.name in the `SecretStore`
- name: conjur
kind: SecretStore
selector:
secret:
# Source Kubernetes secret to be pushed
name: app1-credentials
data:
- match:
# Key in the source Kubernetes secret
secretKey: password
remoteRef:
# <parent policy>/<policy to create>; must contain at least one '/'
remoteKey: data/app1/pushed
# Creates the variable data/app1/pushed/db-password
property: db-password
Save the push secret file as conjur-push-secret.yaml. The source Kubernetes secret (app1-credentials in this
example) must already exist in the same namespace as the PushSecret.
Remote key and policy
In Secrets Manager, a variable must be declared in a policy before a value can be stored in it. For this reason,
ESO does not write values directly to remoteKey. It first loads a policy that declares the variables, and then
sets their values.
The remoteRef.remoteKey must contain at least one /. ESO splits it at the last /:
- the part before the last
/is the parent policy, which must already exist (for exampledata/app1) - the last segment is the name of a policy that ESO creates under the parent (for example
pushed)
On each sync, ESO loads a policy into the parent policy (in POST mode, which only adds to it). The policy contains:
- one variable for each value that is pushed, annotated with
managed-by: external-secrets - a
delegation/consumersgroup, annotated witheditable: "true", which hasreadandexecuteprivileges on those variables. To give an application access to the pushed values, add its host or layer to this group.
The Secrets Manager host that the SecretStore uses must have permission to load policy into the parent policy
(create) and to set the values of the variables that it creates (update). The host also needs read and
execute on those variables, because ESO compares their current values before it writes. The policy grants these
privileges only to delegation/consumers, so add the host to this group if it does not already have them. If ESO
cannot read the value of a variable, it skips that variable without reporting an error, so the variable can keep an
old value while the PushSecret still shows as synced.
Secret key and property
The name of the variable depends on whether secretKey and remoteRef.property are set:
secretKey |
property |
Result |
|---|---|---|
| set | not set | One variable <remoteKey>/<secretKey> that holds the value of that key |
| set | set | One variable <remoteKey>/<property> that holds the value of that key |
| not set | not set | One variable <remoteKey>/<key> for each key in the source secret |
| not set | set | One variable <remoteKey>/<property> that holds all keys of the source secret as a JSON object |
Pushing all keys into one property only works if every value in the source secret is valid UTF-8 text.
Binary values cannot be written into the JSON object correctly.
Existing variables
Before ESO writes a variable, it checks whether the variable already exists:
- If the variable does not exist, ESO creates it.
- If the variable exists but does not have the
managed-by: external-secretsannotation, ESO skips it and does not change it. ESO does not report an error for this. Variables that other tools manage are never overwritten. - If ESO cannot read the current value of the variable, it skips the variable.
- If the value is the same as the value in Kubernetes, ESO skips the variable, so no new version is created.
Update and deletion policies
updatePolicy: IfNotExistsis not supported. Use the defaultReplacepolicy.deletionPolicy: Deleteis not supported. ESO cannot remove variables from Secrets Manager, so deleting the PushSecret returns an error. Removing an entry fromspec.dataalso returns an error, unless the sameremoteKeyis still pushed as a whole secret, orpropertyentries for thatremoteKeyreplace a removed whole-secret entry. In those cases, ESO skips the cleanup and the variables stay in Secrets Manager. Keep the defaultdeletionPolicy: Noneand remove the variables from Secrets Manager yourself.
Create the push secret
# WARNING: creates the push-secret in the "external-secrets" namespace, update the value as needed
#
kubectl apply -n external-secrets -f conjur-push-secret.yaml
# WARNING: running the delete command will delete the push-secrets configuration
# It does not delete the variables in Secrets Manager
#
# If there is a need to delete the push secret
# kubectl delete pushsecret -n external-secrets conjur
Get the Secrets Manager variable
- Check that the push secret is synced:
kubectl get pushsecret -n external-secrets conjur - Log in to your Secrets Manager server and verify that the variable exists and has the
managed-by: external-secretsannotation - Review the value of the variable to verify that it contains the same value as your Kubernetes secret