Rewriting Keys in DataFrom

When calling out an ExternalSecret with dataFrom.extract or dataFrom.find, it is possible that you end up with a kubernetes secret that has conflicts in the key names, or that you simply want to remove a common path from the secret keys.

In order to do so, it is possible to define a set of rewrite operations using dataFrom.rewrite. These operations can be stacked, hence allowing complex manipulations of the secret keys.

Rewrite operations are all applied before ConversionStrategy is applied.

Methods

Regexp

This method implements rewriting through the use of regular expressions. It needs a source and a target field. The source field is where the definition of the matching regular expression goes, where the target field is where the replacing expression goes.

Some considerations about the implementation of Regexp Rewrite:

1. The input of a subsequent rewrite operation are the outputs of the previous rewrite.
2. If a given set of keys do not match any Rewrite operation, there will be no error. Rather, the original keys will be used.
3. If a source is not a compilable regexp expression, an error will be produced and the external secret goes into a error state.

Examples

Removing a common path from find operations

The following ExternalSecret:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: example
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: SecretStore
    name: backend
  target:
    name: secret-to-be-created
  dataFrom:
  - find:
      path: path/to/my
      name: 
        regexp: secrets
    rewrite:
    - regexp:
        source: "path/to/my/secrets/(.*)"
        target: "$1"

Will get all the secrets matching path/to/my/secrets/* and then rewrite them by removing the common path away.

In this example, if we had the following secrets available in the provider:

path/to/my/secrets/username
path/to/my/secrets/password

the output kubernetes secret would be:

apiVersion: v1
kind: Secret
type: Opaque
data:
    username: ...
    password: ...

Avoiding key collisions

The following ExternalSecret:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: example
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: SecretStore
    name: backend
  target:
    name: secret-to-be-created
  dataFrom:
  - extract:
      key: my-secrets-dev
    rewrite:
    - regexp:
        source: "(.*)"
        target: "dev-$1"      
  - extract:
      key: my-secrets-prod
    rewrite:
    - regexp:
        source: "(.*)"
        target: "prod-$1"

Will allow two secrets with the same JSON keys to be imported into a Kubernetes Secret without any conflict. In this example, if we had the following secrets available in the provider:

{
    "my-secrets-dev": {
        "password": "bar",
     },
    "my-secrets-prod": {
        "password": "safebar",
     }
}

the output kubernetes secret would be:

apiVersion: v1
kind: Secret
type: Opaque
data:
    dev_password: YmFy #bar
    prod_password: c2FmZWJhcg== #safebar

Remove invalid characters

The following ExternalSecret:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: example
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: SecretStore
    name: backend
  target:
    name: secret-to-be-created
  dataFrom:
  - extract:
      key: development
    rewrite:
    - regexp:
        source: "[^a-zA-Z0-9 -]"
        target: "_"

Will remove invalid characters from the secret key. In this example, if we had the following secrets available in the provider:

{
    "development": {
        "foo/bar": "1111",
        "foo$baz": "2222"
    }
}

the output kubernetes secret would be:

apiVersion: v1
kind: Secret
type: Opaque
data:
    foo_bar: MTExMQ== #1111
    foo_baz: MjIyMg== #2222

Limitations

Regexp Rewrite is based on golang regexp, which in turns implements RE2 regexp language. There a a series of known limitations to this implementation, such as:

• Lack of ability to do lookaheads or lookbehinds;
• Lack of negation expressions;
• Lack of support for conditional branches;
• Lack of support for possessive repetitions.

A list of compatibility and known limitations considering other commonly used regexp frameworks (such as PCRE and PERL) are listed here.