Skip to content

CyberArk Secrets Manager Provider

This section describes how to set up the CyberArk Secrets Manager provider for External Secrets Operator (ESO). For a working example, see the Accelerator-K8s-External-Secrets repo.

Prerequisites

Before installing the Secrets Manager provider, you need:

  • A running instance of Conjur OSS or CyberArk Secrets Manager, with:
  • An accessible Secrets Manager endpoint (for example: https://myapi.example.com).
  • Your configured Secrets Manager authentication info (such as hostid, apikey or service ID of JWT or Cert authenticator). For more information on configuring Secrets Manager, see Policy statement reference.
  • Support for your authentication method (apikey is supported by default, jwt and cert require additional configuration).
  • Optional: Secrets Manager server certificate (see below).
  • A Kubernetes cluster with ESO installed.

Secrets Manager server certificate

If you set up your Secrets Manager server with a self-signed certificate, we recommend that you populate the caBundle field with the Secrets Manager self-signed certificate in the secret-store definition. The certificate CA must be referenced in the secret-store definition using either caBundle or caProvider:

....
spec:
  provider:
    conjur:
      # Service URL
      url: https://myapi.conjur.org

      # [OPTIONAL] base64 encoded string of certificate
      caBundle: "<base64 encoded cabundle>"

      # [OPTIONAL] caProvider:
      # Instead of caBundle you can also specify a caProvider,
      # which retrieves the cert from a Secret or ConfigMap
      caProvider:
        type: "Secret" # Can be Secret or ConfigMap
        name: "<name of secret or configmap>"
        key: "<key inside secret or configmap>"
        # namespace is required for ClusterSecretStore
        # but not relevant for SecretStore
        namespace: "my-cert-secret-namespace"
  ....

External secret store

To synchronise secrets with Secrets Manager, you must configure an external secret store with one of the following three methods to authenticate to the Secrets Manager API:

  • apikey: uses a Secrets Manager hostid and apikey to authenticate with Secrets Manager
  • jwt: uses a JWT to authenticate with Secrets Manager
  • cert: uses a client certificate and private key to authenticate with Secrets Manager

Option 1: External secret store with apiKey authentication

This method uses a Secrets Manager hostid and apikey to authenticate with Secrets Manager. It is the simplest method to set up and use because your Secrets Manager instance requires no additional configuration.

Step 1: Define an external secret store

Tip

Save as the file as: conjur-secret-store.yaml

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: conjur
spec:
  provider:
    conjur:
      # Service URL
      url: https://myapi.conjur.org
      # [OPTIONAL] base64 encoded string of certificate
      caBundle: OPTIONALxFIELDxxxBase64xCertxString==  
      auth:
        apikey:
          # conjur account
          account: conjur
          userRef: # Get this from K8S secret
            name: conjur-creds
            key: hostid
          apiKeyRef: # Get this from K8S secret
            name: conjur-creds
            key: apikey

Step 2: Create Kubernetes secrets for Secrets Manager credentials

To connect to the Secrets Manager server, the ESO Secrets Manager provider needs to retrieve the apikey credentials from K8s secrets.

Note

For more information about how to create K8s secrets, see Creating a secret.

Here is an example of how to create K8s secrets using the kubectl command:

# This is all one line
kubectl -n external-secrets create secret generic conjur-creds --from-literal=hostid=MYCONJURHOSTID --from-literal=apikey=MYAPIKEY

# Example:
# kubectl -n external-secrets create secret generic conjur-creds --from-literal=hostid=host/data/app1/host001 --from-literal=apikey=321blahblah

Note

conjur-creds is the name defined in the userRef and apikeyRef fields of the conjur-secret-store.yml file.

Step 3: Create the external secrets store

Important

Unless you are using a ClusterSecretStore, credentials must reside in the same namespace as the SecretStore.

# WARNING: creates the store in the "external-secrets" namespace, update the value as needed
#
kubectl apply -n external-secrets -f conjur-secret-store.yaml

# WARNING: running the delete command will delete the secret store configuration
#
# If there is a need to delete the external secretstore
# kubectl delete secretstore -n external-secrets conjur

Option 2: External secret store with JWT authentication

This method uses JWT tokens to authenticate with Secrets Manager. You can use the following methods to retrieve a JWT token for authentication:

  • JWT token from a referenced Kubernetes service account
  • JWT token stored in a Kubernetes secret

Step 1: Define an external secret store

When you use JWT authentication, the following must be specified in the SecretStore:

  • account - The name of the Secrets Manager account
  • serviceId - The ID of the JWT Authenticator WebService configured in Secrets Manager that is used to authenticate the JWT token

You can retrieve the JWT token from either a referenced service account or a Kubernetes secret.

For example, to retrieve a JWT token from a referenced Kubernetes service account, the following secret store definition can be used:

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: conjur
spec:
  provider:
    conjur:
      # Service URL
      url: https://myapi.conjur.org
      # [OPTIONAL] base64 encoded string of certificate
      caBundle: OPTIONALxFIELDxxxBase64xCertxString==
      auth:
        jwt:
          # conjur account
          account: conjur
          # The authn-jwt service ID
          serviceID: my-jwt-auth-service
          # Service account to retrieve JWT token for
          serviceAccountRef:
            name: my-service-account
            # [OPTIONAL] audiences to include in JWT token
            audiences:
              - https://conjur.company.com

Important

This method is only supported in Kubernetes 1.22 and above as it uses the TokenRequest API to get the JWT token from the referenced service account. Audiences can be defined in the Secrets Manager JWT authenticator.

Alternatively, here is an example where a secret containing a valid JWT token is referenced:

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: conjur
spec:
  provider:
    conjur:
      # Service URL
      url: https://myapi.conjur.org
      # [OPTIONAL] base64 encoded string of certificate
      caBundle: OPTIONALxFIELDxxxBase64xCertxString==
      auth:
        jwt:
          # conjur account
          account: conjur
          # The authn-jwt service ID
          serviceID: my-jwt-auth-service
          # Secret containing a valid JWT token
          secretRef:
            name: my-jwt-secret
            key: token

The JWT token must identify your Secrets Manager host, be compatible with your configured Secrets Manager JWT authenticator, and meet all the Secrets Manager JWT guidelines.

You can use an external JWT issuer or the Kubernetes API server to create the token. For example, a Kubernetes service account token can be created with this command:

kubectl create token my-service-account --audience='https://conjur.company.com' --duration=3600s

Save the secret store file as conjur-secret-store.yaml.

Step 2: Create the external secrets store

# WARNING: creates the store in the "external-secrets" namespace, update the value as needed
#
kubectl apply -n external-secrets -f conjur-secret-store.yaml

# WARNING: running the delete command will delete the secret store configuration
#
# If there is a need to delete the external secretstore
# kubectl delete secretstore -n external-secrets conjur

Option 3: External secret store with cert authentication

This method uses a client X.509 certificate and private key to authenticate with Secrets Manager via the authn-cert authenticator. The certificate must be issued by a CA that is trusted by the Secrets Manager authn-cert service. Conjur validates the presented certificate using constraints configured in the authenticator policy (such as cn, san-uri, san-dns, san-ip variables), and optionally per-host annotations that scope constraints to individual workloads.

For more information on configuring the certificate authenticator, see the Secrets Manager Certificate authenticator documentation.

Step 1: Define an external secret store

When you use cert authentication, the following must be specified in the SecretStore:

  • account - The name of the Secrets Manager account
  • serviceID - The ID of the authn-cert WebService configured in Secrets Manager that is used to authenticate the client certificate
  • clientCertRef - Reference to a Kubernetes secret containing the PEM-encoded client certificate
  • clientKeyRef - Reference to a Kubernetes secret containing the PEM-encoded client private key
  • hostId (optional) - The Secrets Manager host identity to authenticate as. Required in the default request mode. Leave empty when using spiffe mode, where the host identity is derived from the SPIFFE SAN URI (spiffe://<trust-domain>/<workload-id>) in the certificate.
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: conjur
spec:
  provider:
    conjur:
      # Service URL
      url: https://myapi.conjur.org
      # [OPTIONAL] base64 encoded string of certificate
      caBundle: OPTIONALxFIELDxxxBase64xCertxString==
      auth:
        cert:
          # conjur account
          account: conjur
          # The authn-cert service ID
          serviceID: my-cert-auth-service
          # [OPTIONAL] HostID for cert authentication (leave empty for 'spiffe' mode).
          # hostId: vm-01
          # Reference to a Kubernetes secret containing the PEM-encoded client certificate
          clientCertRef:
            name: conjur-client-cert
            key: tls.crt
          # Reference to a Kubernetes secret containing the PEM-encoded client key
          clientKeyRef:
            name: conjur-client-cert
            key: tls.key

Step 2: Create Kubernetes secrets for the client certificate and key

To connect to the Secrets Manager server using certificate authentication, the ESO Secrets Manager provider needs to retrieve the client certificate and private key from K8s secrets.

You can use either a TLS-typed secret or a generic secret. The following example uses a generic secret:

# This is all one line
kubectl -n external-secrets create secret generic conjur-client-cert \
  --from-file=tls.crt=/path/to/client.crt \
  --from-file=tls.key=/path/to/client.key

Note

conjur-client-cert is the name defined in the clientCertRef and clientKeyRef fields of the conjur-secret-store.yaml file. Both the certificate and the key must be PEM-encoded.

Step 3: Create the external secrets store

Important

Unless you are using a ClusterSecretStore, credentials must reside in the same namespace as the SecretStore.

# WARNING: creates the store in the "external-secrets" namespace, update the value as needed
#
kubectl apply -n external-secrets -f conjur-secret-store.yaml

# WARNING: running the delete command will delete the secret store configuration
#
# If there is a need to delete the external secretstore
# kubectl delete secretstore -n external-secrets conjur

Option 4: External secret store with IAM authentication

This method uses the AWS IAM authn-iam authenticator to authenticate with Secrets Manager. The ESO pod presents a signed AWS GetCallerIdentity request; Conjur verifies it against the IAM role mapped to the Conjur host. No long-lived credentials are stored — ambient credentials are used automatically (IRSA, EC2 instance profile, ECS task role, environment variables, etc.).

When you use IAM authentication, the following must be specified in the SecretStore:

  • account — The name of the Secrets Manager account.
  • serviceID — The ID of the IAM Authenticator WebService configured in Secrets Manager.
  • hostId — The Conjur host mapped to the AWS IAM role (e.g. data/myapp/123456789012/MyRole).

Optionally, you may reference a Kubernetes Secret that contains explicit AWS credentials via the secretRef field:

  • accessKeyIDSecretRef — Reference to a Secret key containing the AWS Access Key ID.
  • secretAccessKeySecretRef — Reference to a Secret key containing the AWS Secret Access Key.
  • sessionTokenSecretRef (optional) — Reference to a Secret key containing the AWS Session Token. Required only when using temporary credentials (for example, credentials obtained via AssumeRole or AssumeRoleWithWebIdentity).

If secretRef is omitted, the default AWS SDK credential chain is used.

The Secrets Manager url must use https:// when using IAM authentication, since the AWS IAM authenticator sends a signed request as a bearer credential. If you must connect over plain HTTP (for example, a service mesh sidecar that terminates TLS locally), set insecure: true to explicitly opt out of this check.

Step 1: Define an external secret store

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: conjur-iam
  namespace: default
spec:
  provider:
    conjur:
      url: https://conjur.example.com
      # caBundle: <base64-encoded PEM>  # uncomment for self-signed certs
      auth:
        iam:
          account: myorg
          serviceID: prod          # authn-iam webservice ID in Conjur policy
          hostId: data/myapp/123456789012/MyRole  # Conjur host mapped to the AWS IAM role
          # secretRef:             # omit to use ambient credentials (IRSA, instance profile, etc.)
          #   accessKeyIDSecretRef:
          #     name: aws-creds
          #     key: access-key-id
          #   secretAccessKeySecretRef:
          #     name: aws-creds
          #     key: secret-access-key
          #   sessionTokenSecretRef:  # required only when using temporary credentials
          #     name: aws-creds
          #     key: session-token
          # insecure: true          # required to authenticate over a plain HTTP url; use only when the connection is otherwise protected

Step 2: Create the external secrets store

kubectl apply -n external-secrets -f conjur-secret-store.yaml

Option 5: External secret store with Azure authentication

This method uses the Azure authn-azure authenticator to authenticate with Secrets Manager. The ESO pod presents an Azure JWT token; Conjur verifies it against the managed identity mapped to the Conjur host.

Two sub-modes are available:

  • Ambient IMDS (recommended on AKS / Azure VM): no serviceAccountRef is set. conjur-api-go fetches the Azure JWT from the Azure Instance Metadata Service (IMDS) automatically. Use this when the pod runs on Azure infrastructure with a managed identity attached.
  • ServiceAccount token (serviceAccountRef): ESO requests a Kubernetes ServiceAccount token via the TokenRequest API and forwards it as the Azure JWT. Use this for local testing or when running outside Azure.

For user-assigned managed identities you must also set clientId to the client ID of the managed identity. For system-assigned identities, omit clientId.

Step 1: Define an external secret store (ambient IMDS)

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: conjur-azure-ambient
  namespace: default
spec:
  provider:
    conjur:
      url: https://conjur.example.com
      # caBundle: <base64-encoded PEM>  # uncomment for self-signed certs
      auth:
        azure:
          account: myorg
          serviceID: prod          # authn-azure webservice ID in Conjur policy
          hostId: data/myapp/myhost  # Conjur host mapped to the Azure managed identity
          # clientId: <managed-identity-client-id>  # required for user-assigned managed identities

Step 1 (alternative): Define an external secret store (ServiceAccount token)

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: conjur-azure-sa
  namespace: default
spec:
  provider:
    conjur:
      url: https://conjur.example.com
      # caBundle: <base64-encoded PEM>  # uncomment for self-signed certs
      auth:
        azure:
          account: myorg
          serviceID: prod
          hostId: data/myapp/myhost
          serviceAccountRef:
            name: my-service-account  # Kubernetes ServiceAccount in the same namespace
            audiences:
              - conjur              # must match the Conjur authn-azure policy audience

Step 2: Create the external secrets store

kubectl apply -n external-secrets -f conjur-secret-store.yaml

Option 6: External secret store with GCP authentication

This method uses the GCP authn-gcp authenticator to authenticate with Secrets Manager. The ESO pod presents a GCP identity token; Conjur verifies it against the service account mapped to the Conjur host.

Two sub-modes are supported:

  • Ambient GCP Metadata Service (recommended on GKE with Workload Identity): no secretRef is set. conjur-api-go fetches the GCP identity token from the GCP Metadata Service automatically. Use this when the pod runs on GKE with Workload Identity or on a GCE instance.
  • Kubernetes Secret (secretRef): a pre-obtained GCP identity token is read from a Kubernetes Secret. Use this for local testing or when running outside GCP.

The following must be specified in the SecretStore:

  • account — The Conjur organization account name.
  • hostId — The Conjur host mapped to the GCP service account (e.g. data/myapp/myhost).

Optionally, you may set serviceID to the GCP Authenticator WebService ID configured in Conjur policy. This field is reserved for future use; Conjur's authn-gcp authenticator does not include the service ID in the authentication URL.

Step 1: Define an external secret store (ambient GCP Metadata Service)

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: conjur-gcp-ambient
  namespace: default
spec:
  provider:
    conjur:
      url: https://conjur.example.com
      # caBundle: <base64-encoded PEM>  # uncomment for self-signed certs
      auth:
        gcp:
          account: myorg
          serviceID: prod          # reserved for future use; not used by authn-gcp today
          hostId: data/myapp/myhost  # Conjur host mapped to the GCP service account

Step 1 (alternative): Define an external secret store (explicit token from Kubernetes Secret)

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: conjur-gcp-secretref
  namespace: default
spec:
  provider:
    conjur:
      url: https://conjur.example.com
      # caBundle: <base64-encoded PEM>  # uncomment for self-signed certs
      auth:
        gcp:
          account: myorg
          serviceID: prod          # reserved for future use; not used by authn-gcp today
          hostId: data/myapp/myhost
          secretRef:
            jwt:
              name: gcp-token-secret  # Kubernetes Secret in the same namespace
              key: token              # key within the Secret holding the GCP identity token

Step 2: Create the external secrets store

kubectl apply -n external-secrets -f conjur-secret-store.yaml

Define an external secret

After you have configured the Secrets Manager provider secret store, you can fetch secrets from Secrets Manager.

Here is an example of how to fetch a single secret from Secrets Manager:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: conjur
spec:
  refreshInterval: 1h0m0s
  secretStoreRef:
    # This name must match the metadata.name in the `SecretStore`
    name: conjur
    kind: SecretStore
  data:
  - secretKey: secret00
    remoteRef:
      key: data/app1/secret00

Save the external secret file as conjur-external-secret.yaml.

Find by Name or Find by Tag

The Secrets Manager provider also supports the Find by Name and Find by Tag ESO features. This means that you can use a regular expression or tags to dynamically fetch multiple secrets from Secrets Manager.

These two filters are mutually exclusive on a single find: if both name and tags are set, only name is used and tags are ignored (see issue #6554).

To find secrets by name:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: conjur-find-by-name
spec:
  refreshInterval: 1h0m0s
  secretStoreRef:
    # This name must match the metadata.name in the `SecretStore`
    name: conjur
    kind: SecretStore
  target:
    name: k8s-secret-to-be-created
  dataFrom:
    - find:
        name:
          # Match all secrets in the app1 namespace (e.g., `app1/secret00`, `app1/secret01`, etc.)
          regexp: "^app1\/.+$"

To find secrets by tag, use tags instead of name:

  dataFrom:
    - find:
        tags:
          environment: "prod"
          application: "app1"

If you use these features, we strongly recommend that you limit the permissions of the Secrets Manager host to only the secrets that it needs to access. This is more secure and it reduces the load on both the Secrets Manager server and ESO.

Create the external secret

# WARNING: creates the external-secret in the "external-secrets" namespace, update the value as needed
#
kubectl apply -n external-secrets -f conjur-external-secret.yaml

# WARNING: running the delete command will delete the external-secrets configuration
#
# If there is a need to delete the external secret
# kubectl delete externalsecret -n external-secrets conjur

Get the K8s secret

  • Log in to your Secrets Manager server and verify that your secret exists
  • Review the value of your Kubernetes secret to verify that it contains the same value as the Secrets Manager server
# WARNING: this command will reveal the stored secret in plain text
#
# Assuming the secret name is "secret00", this will show the value
kubectl get secret -n external-secrets conjur -o jsonpath="{.data.secret00}"  | base64 --decode && echo

Define a push secret

The Secrets Manager provider also supports PushSecret. This means that you can push the values of a Kubernetes secret into Secrets Manager variables.

Here is an example of how to push a single key of a Kubernetes secret into Secrets Manager:

apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
  name: conjur
spec:
  refreshInterval: 1h0m0s
  # Only the Replace update policy is supported by the Secrets Manager provider.
  updatePolicy: Replace
  # Deleting pushed variables is not supported, keep the default deletion policy.
  deletionPolicy: None
  secretStoreRefs:
    # This name must match the metadata.name in the `SecretStore`
    - name: conjur
      kind: SecretStore
  selector:
    secret:
      # Source Kubernetes secret to be pushed
      name: app1-credentials
  data:
    - match:
        # Key in the source Kubernetes secret
        secretKey: password
        remoteRef:
          # <parent policy>/<policy to create>; must contain at least one '/'
          remoteKey: data/app1/pushed
          # Creates the variable data/app1/pushed/db-password
          property: db-password

Save the push secret file as conjur-push-secret.yaml. The source Kubernetes secret (app1-credentials in this example) must already exist in the same namespace as the PushSecret.

Remote key and policy

In Secrets Manager, a variable must be declared in a policy before a value can be stored in it. For this reason, ESO does not write values directly to remoteKey. It first loads a policy that declares the variables, and then sets their values.

The remoteRef.remoteKey must contain at least one /. ESO splits it at the last /:

  • the part before the last / is the parent policy, which must already exist (for example data/app1)
  • the last segment is the name of a policy that ESO creates under the parent (for example pushed)

On each sync, ESO loads a policy into the parent policy (in POST mode, which only adds to it). The policy contains:

  • one variable for each value that is pushed, annotated with managed-by: external-secrets
  • a delegation/consumers group, annotated with editable: "true", which has read and execute privileges on those variables. To give an application access to the pushed values, add its host or layer to this group.

The Secrets Manager host that the SecretStore uses must have permission to load policy into the parent policy (create) and to set the values of the variables that it creates (update). The host also needs read and execute on those variables, because ESO compares their current values before it writes. The policy grants these privileges only to delegation/consumers, so add the host to this group if it does not already have them. If ESO cannot read the value of a variable, it skips that variable without reporting an error, so the variable can keep an old value while the PushSecret still shows as synced.

Secret key and property

The name of the variable depends on whether secretKey and remoteRef.property are set:

secretKey property Result
set not set One variable <remoteKey>/<secretKey> that holds the value of that key
set set One variable <remoteKey>/<property> that holds the value of that key
not set not set One variable <remoteKey>/<key> for each key in the source secret
not set set One variable <remoteKey>/<property> that holds all keys of the source secret as a JSON object

Pushing all keys into one property only works if every value in the source secret is valid UTF-8 text. Binary values cannot be written into the JSON object correctly.

Existing variables

Before ESO writes a variable, it checks whether the variable already exists:

  • If the variable does not exist, ESO creates it.
  • If the variable exists but does not have the managed-by: external-secrets annotation, ESO skips it and does not change it. ESO does not report an error for this. Variables that other tools manage are never overwritten.
  • If ESO cannot read the current value of the variable, it skips the variable.
  • If the value is the same as the value in Kubernetes, ESO skips the variable, so no new version is created.

Update and deletion policies

  • updatePolicy: IfNotExists is not supported. Use the default Replace policy.
  • deletionPolicy: Delete is not supported. ESO cannot remove variables from Secrets Manager, so deleting the PushSecret returns an error. Removing an entry from spec.data also returns an error, unless the same remoteKey is still pushed as a whole secret, or property entries for that remoteKey replace a removed whole-secret entry. In those cases, ESO skips the cleanup and the variables stay in Secrets Manager. Keep the default deletionPolicy: None and remove the variables from Secrets Manager yourself.

Create the push secret

# WARNING: creates the push-secret in the "external-secrets" namespace, update the value as needed
#
kubectl apply -n external-secrets -f conjur-push-secret.yaml

# WARNING: running the delete command will delete the push-secrets configuration
# It does not delete the variables in Secrets Manager
#
# If there is a need to delete the push secret
# kubectl delete pushsecret -n external-secrets conjur

Get the Secrets Manager variable

  • Check that the push secret is synced: kubectl get pushsecret -n external-secrets conjur
  • Log in to your Secrets Manager server and verify that the variable exists and has the managed-by: external-secrets annotation
  • Review the value of the variable to verify that it contains the same value as your Kubernetes secret

See also