Skip to content

Security Best Practices

The purpose of this document is to provide a set of best practices for securing External Secrets Operator. These best practices are designed to reduce the risk of a successful attack against the operator or the Kubernetes cluster it integrates with.

Pod Security

The External Secrets Operator Pods have been configured to adhere to Pod Security Standards restricted profile which establish a great security posture by following current Pod hardening best practices such as the NSA Kubernetes Hardening Guide.

These measures provide a secure and robust environment for the External Secrets Operator to operate within. They have been set as default since v0.8.2 and should not be changed to conform with the principle of least privilege.

RBAC

The External Secrets Operator runs with highly privileged access to your Kubernetes cluster. Due to it's purpose it is able to read and write to all secrets across all namespaces.

Make sure to restrict access to ESO resources like ExternalSecret, SecretStore etc. where appropriate. This is particularly important for cluster-scoped resources like ClusterExternalSecret and ClusterSecretStore. If an attacker is able to tamper with these resources he could potentially get unauthorized access to secrets or may be able to exfiltrate data out of your system.

In most scenarios the External Secrets Operator runs cluster-wide. If this is not desired and you want to run it per-namespace, you can scope it to a particular namespace, see scopedRBAC and scopedNamespace in the helm chart.

A short checklist for you to walk through:

  • restrict access to execute a shell pods/exec in External Secrets Operator Pod
  • restrict access to (Cluster)ExternalSecret and (Cluster)SecretStore
  • restrict access to aggregated ClusterRoles view/edit/admin where needed
  • run ESO with scoped RBAC/Namespace if needed

Network Policy

Network Traffic from/to the operator should be restricted using NetworkPolicies or similar. By default, External Secrets Operator does not provide Network Policies for you.

Data Exfiltration Risk

If not configured properly ESO may be used to exfiltrate data out of your cluster. It is advised to create tight NetworkPolicies and use a policy engine such as kyverno to prevent data exfiltration.

You should restrict access in egress direction:

  • controller
    • kube-apiserver
    • secret provider (AWS, GCP, ...), where possible use private endpoints.
  • webhook
    • kube-apiserver
  • cert-controller
    • kube-apiserver

Further, you also should restrict ingress traffic to ESO Pods:

  • controller
    • :8080 from you monitoring agent
  • cert-controller:
    • :8080 from you monitoring agent
    • :8081 from kubelet (healthz/readyz)
  • webhook:
    • :10250 from the kube-apiserver
    • :8080 from you monitoring agent
    • :8081 from kubelet (healthz/readyz)

Policy Engine Best Practices

You should use a policy engine like kyverno or OPA Gatekeeper to restrict changes made to ESO resources like SecretStore and ExternalSecret.

Data Exfiltration Risk

ESO could be used to exfiltrate data out of your cluster. You should disable all providers you don't need. Further, you should implement NetworkPolicies to restrict network access to known entities (see above), to prevent data exfiltration.

Here a couple of recommendations for you to consider:

  • explicitly deny usage of the providers you don't need
  • restrict access to secrets with/without a particular prefix in .spec.data[].remoteRef.key
  • restrict usage of a (Cluster)SecretStore reference in an ExternalSecret

Provider Validation Example Policy

The following policy validates the usage of the provider field in the SecretStore manifest.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-secretstore-aws-provider
spec:
  validationFailureAction: Enforce
  rules:
  - name: require-secretstore-aws-provider
    match:
      any:
      - resources:
          kinds:
          - SecretStore
          - ClusterSecretStore
    validate:
      message: "You must only use AWS SecretsManager"
      pattern:
        spec:
          provider:
            aws:
              service: SecretsManager

Regular Patches

Regularly patch and update all software components of ESO and the cluster to ensure that known vulnerabilities are addressed. Use automated patching and updating tools to ensure that all components are kept up-to-date.

We provide regular updates to ESO, see Stability and Support.

Verify Artefacts

Verify Container Images

External Secrets Operator container images are signed using Cosign and the keyless signing feature. To verify the container image, follow the steps below.

$ crane digest ghcr.io/external-secrets/external-secrets:v0.8.1
sha256:36e606279dbebac51b4b9300b9fa85e8c08c1c673ba3ecc38af1402a0b035554

$ COSIGN_EXPERIMENTAL=1 cosign verify ghcr.io/external-secrets/external-secrets@sha256:36e606279dbebac51b4b9300b9fa85e8c08c1c673ba3ecc38af1402a0b035554 | jq

# ...
[
  {
    "critical": {
      "identity": {
        "docker-reference": "ghcr.io/external-secrets/external-secrets"
      },
      "image": {
        "docker-manifest-digest": "sha256:36e606279dbebac51b4b9300b9fa85e8c08c1c673ba3ecc38af1402a0b035554"
      },
      "type": "cosign container image signature"
    },
    "optional": {
      "1.3.6.1.4.1.57264.1.1": "https://token.actions.githubusercontent.com",
      "1.3.6.1.4.1.57264.1.2": "workflow_dispatch",
      "1.3.6.1.4.1.57264.1.3": "a0d2aef2e35c259c9ee75d65f7587e6ed71ef2ad",
      "1.3.6.1.4.1.57264.1.4": "Create Release",
      "1.3.6.1.4.1.57264.1.5": "external-secrets/external-secrets",
      "1.3.6.1.4.1.57264.1.6": "refs/heads/main",
      "Bundle": {
        # ...
      },
      "GITHUB_ACTOR": "gusfcarvalho",
      "Issuer": "https://token.actions.githubusercontent.com",
      "Subject": "https://github.com/external-secrets/external-secrets/.github/workflows/release.yml@refs/heads/main",
      "githubWorkflowName": "Create Release",
      "githubWorkflowRef": "refs/heads/main",
      "githubWorkflowRepository": "external-secrets/external-secrets",
      "githubWorkflowSha": "a0d2aef2e35c259c9ee75d65f7587e6ed71ef2ad",
      "githubWorkflowTrigger": "workflow_dispatch"
    }
  }
]

Note that the important fields to verify in the output are optional.Issuer and optional.Subject. If Issuer and Subject do not match the values shown above, the image is not legit and should not be used.

Verify Provenance

External Secrets Operator creates and attests to the provenance of its builds using the SLSA standard. The attested provenance may be verified using the cosign tool.

$ COSIGN_EXPERIMENTAL=1 cosign verify-attestation --type slsaprovenance ghcr.io/external-secrets/external-secrets:v0.8.1 | jq .payload -r | base64 --decode | jq

Verification for ghcr.io/external-secrets/external-secrets:v0.8.1 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - Any certificates were verified against the Fulcio roots.
Certificate subject:  https://github.com/external-secrets/external-secrets/.github/workflows/release.yml@refs/heads/main
Certificate issuer URL:  https://token.actions.githubusercontent.com
GitHub Workflow Trigger: workflow_dispatch
GitHub Workflow SHA: a0d2aef2e35c259c9ee75d65f7587e6ed71ef2ad
GitHub Workflow Name: Create Release
GitHub Workflow Trigger external-secrets/external-secrets
GitHub Workflow Ref: refs/heads/main
{
  "_type": "https://in-toto.io/Statement/v0.1",
  "predicateType": "https://slsa.dev/provenance/v0.2",
  "subject": [
    {
      "name": "ghcr.io/external-secrets/external-secrets",
      "digest": {
        "sha256": "36e606279dbebac51b4b9300b9fa85e8c08c1c673ba3ecc38af1402a0b035554"
      }
    }
  ],
  "predicate": {
    "builder": {
      "id": "https://github.com/external-secrets/external-secrets/Attestations/GitHubHostedActions@v1"
    },
    "buildType": "https://github.com/Attestations/GitHubActionsWorkflow@v1",
    "invocation": {
      "configSource": {
        "uri": "git+https://github.com/external-secrets/external-secrets",
        "digest": {
          "sha1": "a0d2aef2e35c259c9ee75d65f7587e6ed71ef2ad"
        },
        "entryPoint": "Create Release"
      },
      "parameters": {
        "version": "v0.8.1"
      }
    },
    [...]
  }
}

Fetching SBOM

An SBOM (Software Bill of Materials) in Software Package Data Exchange (SPDX) JSON format is attached to every External Secrets Operator image. To download and verify the SBOM for a specific version, install Cosign and run:

$ crane digest ghcr.io/external-secrets/external-secrets:v0.8.1
sha256:36e606279dbebac51b4b9300b9fa85e8c08c1c673ba3ecc38af1402a0b035554

$ COSIGN_EXPERIMENTAL=1 cosign verify-attestation --type spdx ghcr.io/external-secrets/external-secrets@sha256:36e606279dbebac51b4b9300b9fa85e8c08c1c673ba3ecc38af1402a0b035554 | jq '.payload |= @base64d | .payload | fromjson' | jq '.predicate.Data | fromjson'

[...]
{
  "SPDXID": "SPDXRef-DOCUMENT",
  "name": "ghcr.io/external-secrets/external-secrets@sha256-36e606279dbebac51b4b9300b9fa85e8c08c1c673ba3ecc38af1402a0b035554",
  "spdxVersion": "SPDX-2.2",
  "creationInfo": {
    "created": "2023-03-17T23:17:01.568002344Z",
    "creators": [
      "Organization: Anchore, Inc",
      "Tool: syft-0.40.1"
    ],
    "licenseListVersion": "3.16"
  },
  "dataLicense": "CC0-1.0",
  "documentNamespace": "https://anchore.com/syft/image/ghcr.io/external-secrets/external-secrets@sha256-36e606279dbebac51b4b9300b9fa85e8c08c1c673ba3ecc38af1402a0b035554-83484ebb-b469-45fa-8fcc-9290c4ea4f6f",
  "packages": [
    [...]
    {
      "SPDXID": "SPDXRef-c809070b0beb099e",
      "name": "tzdata",
      "licenseConcluded": "NONE",
      "downloadLocation": "NOASSERTION",
      "externalRefs": [
        {
          "referenceCategory": "SECURITY",
          "referenceLocator": "cpe:2.3:a:tzdata:tzdata:2021a-1\\+deb11u8:*:*:*:*:*:*:*",
          "referenceType": "cpe23Type"
        },
        {
          "referenceCategory": "PACKAGE_MANAGER",
          "referenceLocator": "pkg:deb/debian/tzdata@2021a-1+deb11u8?arch=all&distro=debian-11",
          "referenceType": "purl"
        }
      ],
      "filesAnalyzed": false,
      "licenseDeclared": "NONE",
      "originator": "Person: GNU Libc Maintainers <debian-glibc@lists.debian.org>",
      "sourceInfo": "acquired package info from DPKG DB: /var/lib/dpkg/status.d/tzdata, /usr/share/doc/tzdata/copyright",
      "versionInfo": "2021a-1+deb11u8"
    }
  ]
}