Skip to content

AWS Secrets Manager

aws sm

Secrets Manager

A SecretStore points to AWS Secrets Manager in a certain account within a defined region. You should define Roles that define fine-grained access to individual secrets and pass them to ESO using spec.provider.aws.role. This way users of the SecretStore can only access the secrets necessary.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secretsmanager
spec:
  provider:
    aws:
      service: SecretsManager
      # define a specific role to limit access
      # to certain secrets.
      # role is a optional field that
      # can be omitted for test purposes
      role: arn:aws:iam::123456789012:role/external-secrets
      region: eu-central-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: awssm-secret
            key: access-key
          secretAccessKeySecretRef:
            name: awssm-secret
            key: secret-access-key
NOTE: In case of a ClusterSecretStore, Be sure to provide namespace in accessKeyIDSecretRef and secretAccessKeySecretRef with the namespaces where the secrets reside.

NOTE: When using dataFrom without a path defined, the provider will fall back to using ListSecrets. ListSecrets then proceeds to fetch each individual secret in turn. To use BatchGetSecretValue and avoid excessive API calls define a path prefix or use Tags filter.

IAM Policy

Create a IAM Policy to pin down access to secrets matching dev-*.

For Batch permissions read the following post https://aws.amazon.com/about-aws/whats-new/2023/11/aws-secrets-manager-batch-retrieval-secrets/.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action" : [
        "secretsmanager:ListSecrets",
        "secretsmanager:BatchGetSecretValue"
      ],
      "Effect" : "Allow",
      "Resource" : "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetResourcePolicy",
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret",
        "secretsmanager:ListSecretVersionIds"
      ],
      "Resource": [
        "arn:aws:secretsmanager:us-west-2:111122223333:secret:dev-*"
      ]
    }
  ]
}

Permissions for PushSecret

If you're planning to use PushSecret, ensure you also have the following permissions in your IAM policy:

{
  "Effect": "Allow",
  "Action": [
    "secretsmanager:CreateSecret",
    "secretsmanager:PutSecretValue",
    "secretsmanager:TagResource",
    "secretsmanager:DeleteSecret"
  ],
  "Resource": [
    "arn:aws:secretsmanager:us-west-2:111122223333:secret:dev-*"
  ]
}

Here's a more restrictive version of the IAM policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:CreateSecret",
        "secretsmanager:PutSecretValue",
        "secretsmanager:TagResource"
      ],
      "Resource": [
        "arn:aws:secretsmanager:us-west-2:111122223333:secret:dev-*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:DeleteSecret"
      ],
      "Resource": [
        "arn:aws:secretsmanager:us-west-2:111122223333:secret:dev-*"
      ],
      "Condition": {
        "StringEquals": {
          "secretsmanager:ResourceTag/managed-by": "external-secrets"
        }
      }
    }
  ]
}

In this policy, the DeleteSecret action is restricted to secrets that have the specified tag, ensuring that deletion operations are more controlled and in line with the intended management of the secrets.

Additional Settings for PushSecret

Additional settings can be set at the SecretStore level to control the behavior of PushSecret when interacting with AWS Secrets Manager.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secretsmanager
spec:
  provider:
    aws:
      service: SecretsManager
      role: arn:aws:iam::123456789012:role/external-secrets
      region: eu-central-1
      secretsManager:
        # Additional parameters can be added to the AWS Secrets Manager DeleteSecret API call.
        # These parameters are only relevant when the deletionPolicy is set to Delete.
        # See: https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_DeleteSecret.html#API_DeleteSecret_RequestSyntax
        forceDeleteWithoutRecovery: true
        # recoveryWindowInDays: 9 (conflicts with forceDeleteWithoutRecovery)

Additional Metadata for PushSecret

Optionally, it is possible to configure additional options for the parameter. These are as follows: - kmsKeyID - secretPushFormat - description - tags

To control this behavior set the following provider metadata:

apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
  name: pushsecret-example # Customisable
  namespace: teamb # Same of the SecretStores
spec:
  deletionPolicy: Delete
  refreshInterval: 1h # Refresh interval for which push secret will reconcile
  secretStoreRefs: # A list of secret stores to push secrets to
    - name: teamb-secret-store
      kind: SecretStore
  selector:
    secret:
      name: my-secret # Source Kubernetes secret to be pushed
  data:
    - match:
        secretKey: key1 # Source Kubernetes secret key to be pushed
        remoteRef:
          remoteKey: teamb-my-first-parameter-3 # Remote reference (where the secret is going to be pushed)
      metadata:
        apiVersion: kubernetes.external-secrets.io/v1alpha1
        kind: PushSecretMetadata
        spec:
          kmsKeyID: bb123123-b2b0-4f60-ac3a-44a13f0e6b6c # When not set, default to alias/aws/secretsmanager
          secretPushFormat: string # When not set, default to binary
          description: "secret 'managed-by:secret-manager' from 'secret-store:teamb-secret-store'"
          tags:
            secret-store: teamb-secret-store
            refresh-interval: 1h
  • secretPushFormat takes two options. binary and string, where binary is the default.
  • kmsKeyID takes a KMS Key $ID or $ARN (in case a key source is created in another account) as a string, where alias/aws/secretsmanager is the default.
  • description Description of the secret.
  • tags Key-value map of user-defined tags that are attached to the secret.

JSON Secret Values

SecretsManager supports simple key/value pairs that are stored as json. If you use the API you can store more complex JSON objects. You can access nested values or arrays using gjson syntax:

Consider the following JSON object that is stored in the SecretsManager key friendslist: 

{
  "name": {"first": "Tom", "last": "Anderson"},
  "friends": [
    {"first": "Dale", "last": "Murphy"},
    {"first": "Roger", "last": "Craig"},
    {"first": "Jane", "last": "Murphy"}
  ]
}

This is an example on how you would look up nested keys in the above json object:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
    kind: SecretStore
  target:
    name: friends
    creationPolicy: Owner
  data:
  - secretKey: my_name
    remoteRef:
      key: friendslist
      property: name.first # Tom
  - secretKey: first_friend
    remoteRef:
      key: friendslist
      property: friends.1.first # Roger

  # metadataPolicy to fetch all the labels in JSON format
  - secretKey: tags
    remoteRef:
      metadataPolicy: Fetch 
      key: database-credentials

  # metadataPolicy to fetch a specific label (dev) from the source secret
  - secretKey: developer
    remoteRef:
      metadataPolicy: Fetch 
      key: database-credentials
      property: dev

Secret Versions

SecretsManager creates a new version of a secret every time it is updated. The secret version can be reference in two ways, the VersionStage and the VersionId. The VersionId is a unique uuid which is generated every time the secret changes. This id is immutable and will always refer to the same secret data. The VersionStage is an alias to a VersionId, and can refer to different secret data as the secret is updated. By default, SecretsManager will add the version stages AWSCURRENT and AWSPREVIOUS to every secret, but other stages can be created via the update-secret-version-stage api.

The version field on the remoteRef of the ExternalSecret will normally consider the version to be a VersionStage, but if the field is prefixed with uuid/, then the version will be considered a VersionId.

So in this example, the operator will request the same secret with different versions: AWSCURRENT and AWSPREVIOUS:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: versioned-api-key
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
    kind: SecretStore
  target:
    name: versioned-api-key
    creationPolicy: Owner
  data:
  - secretKey: previous-api-key
    remoteRef:
      key: "production/api-key"
      version: "AWSPREVIOUS"
  - secretKey: current-api-key
    remoteRef:
      key: "production/api-key"
      version: "AWSCURRENT"

While in this example, the operator will request the secret with VersionId as abcd-1234

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: versioned-api-key
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
    kind: SecretStore
  target:
    name: versioned-api-key
    creationPolicy: Owner
  data:
  - secretKey: api-key
    remoteRef:
      key: "production/api-key"
      version: "uuid/123e4567-e89b-12d3-a456-426614174000"

AWS Authentication

Controller's Pod Identity

Pod Identity Authentication

Note: If you are using Parameter Store replace service: SecretsManager with service: ParameterStore in all examples below.

This is basically a zero-configuration authentication method that inherits the credentials from the runtime environment using the aws sdk default credential chain.

You can attach a role to the pod using IRSA, kiam or kube2iam. When no other authentication method is configured in the Kind=Secretstore this role is used to make all API calls against AWS Secrets Manager or SSM Parameter Store.

Based on the Pod's identity you can do a sts:assumeRole before fetching the secrets to limit access to certain keys in your provider. This is optional.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: team-b-store
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-central-1
      # optional: do a sts:assumeRole before fetching secrets
      role: team-b

Access Key ID & Secret Access Key

SecretRef

You can store Access Key ID & Secret Access Key in a Kind=Secret and reference it from a SecretStore.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: team-b-store
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-central-1
      # optional: assume role before fetching secrets
      role: team-b
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: awssm-secret
            key: access-key
          secretAccessKeySecretRef:
            name: awssm-secret
            key: secret-access-key

NOTE: In case of a ClusterSecretStore, Be sure to provide namespace in accessKeyIDSecretRef, secretAccessKeySecretRef with the namespaces where the secrets reside.

EKS Service Account credentials

Service Account

This feature lets you use short-lived service account tokens to authenticate with AWS. You must have Service Account Volume Projection enabled - it is by default on EKS. See EKS guide on how to set up IAM roles for service accounts.

The big advantage of this approach is that ESO runs without any credentials.

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/team-a
  name: my-serviceaccount
  namespace: default

Reference the service account from above in the Secret Store:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: secretstore-sample
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-central-1
      auth:
        jwt:
          serviceAccountRef:
            name: my-serviceaccount

NOTE: In case of a ClusterSecretStore, Be sure to provide namespace for serviceAccountRef with the namespace where the service account resides.

EKS Pod Identity Setup

In order to use EKS Pod Identity Agent, create a role like this:

{
    "Statement": [
        {
            "Action": [
                "secretsmanager:GetResourcePolicy",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecretVersionIds"
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        }
    ],
    "Version": "2012-10-17"
}

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
            "Effect": "Allow",
            "Principal": {
                "Service": "pods.eks.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession"
            ]
        }
    ]
}

Install ESO using helm and define these values:

serviceAccount:
  annotations:
  name: external-secrets

Create a pod association:

aws eks create-pod-identity-association --cluster-name my-cluster --role-arn arn:aws:iam::111122223333:role/my-role --namespace external-secrets --service-account external-secrets

Then create a secret store like this:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: store
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-central-1

Note: serviceAccountRef cannot be used together with EKS Pod Identity. That's because ESO can not impersonate service accounts which have iam roles bound using pod identity. Doing so will result in an error like this: 

unable to create session: an IAM role must be associated with service account ...

Note: No auth section is defined for the SecretStore.

Note: For even more details you can follow this post for more setup and information using Terraform here.

Custom Endpoints

You can define custom AWS endpoints if you want to use regional, vpc or custom endpoints. See List of endpoints for Secrets Manager, Secure Systems Manager and Security Token Service.

Use the following environment variables to point the controller to your custom endpoints. Note: All resources managed by this controller are affected.

ENV VAR DESCRIPTION
AWS_SECRETSMANAGER_ENDPOINT Endpoint for the Secrets Manager Service. The controller uses this endpoint to fetch secrets from AWS Secrets Manager.
AWS_SSM_ENDPOINT Endpoint for the AWS Secure Systems Manager. The controller uses this endpoint to fetch secrets from SSM Parameter Store.
AWS_STS_ENDPOINT Endpoint for the Security Token Service. The controller uses this endpoint when creating a session and when doing assumeRole or assumeRoleWithWebIdentity calls.