Kubernetes Secrets from Secrets Manager using External Secrets Operators

7 min readMar 26, 2022

In this article we will demonstrate how we store our credentials in the Secrets Manager and automatically retrieve them for creating Kubernetes Secrets. The External Secrets is the tool that we will use in this article.

Several times as a DevOps, I discover that the applications that we deployed to Kubernetes usually need some kinds of credentials to pass into the applications. Yes, Kubernetes provides the way to abstract the confidential information into its own resource, it is the Secret. That means the Secrets need to be in placed before we deploy our applications.

In real life, there are many applications that DevOps need to maintain and it’s possible that the same applications need to be deployed to many environments (dev, test, prod). This is really the nightmare for DevOps to operate on the situation like that.

If you’re the persons that found the situation like I described above and want to solve it. Yes, the External Secret is your good friend.

If you are thinking about hard code the secrets in the source code and put them in the application’s package and believe that this way will solve the issuer. Don’t do this, it’s not the right approach.

Prerequisites

This article won’t go too much in details about the technology stacks we will use. We assume that you should have some knowledges of the following.

Helm — We will use it to deploy the External Secrets Operator on GKE.
Secrets Manager — We will store the secrets in the Secrets Manager
Kubernetes.

Source Code

The source codes relate to this article are kept publicly in the GitHub repository here — https://github.com/its-knowledge-sharing/setup-gke-external-secret. I would recommend to clone the code and read them at the same time you read this article.

Create the GKE cluster

I’m assuming that we are familiar to the Google Cloud Platform (GCP) and already have the GCP account. See this script for more detail 01-setup-gke.bash, run the script and wait until the GKE is created.

Create service account (SA)

To allow External Secrets to retrieve the secrets, we will need to provide the SA for External Secrets to talk to Secrets Manager. See this script for more detail 02-create-sa.bash.

In the script above, we granted the role roles/secretmanager.secretAccessor to the service account, which is enough for External Secrets to use.

Deploy External Secrets to GKE

The easiest way we deploy External Secrets into Kubernetes is by using Helm. The External Secrets also provided it’s own Helm chart here https://charts.external-secrets.io/. What we need to do is to create the Helm values file to customize what we need and run some Helm commands to get everything.

To simplify things, I wrote a Secrets Manager deployment script which can be viewed here 03–1-deploy-external-secret.bash. If everything is OK, we should see the result as shown below. Please note that there are 3 Custom Resource Definition (CRD) created which later we will use 2 of them.

The script also created the Kubernetes Secret gcp-secret-manager to store the GCP service account that External Secrets need to use.

Cluster Secret Store

As mentioned earlier that there are 3 CRDs created when we deploy External Secrets by using the script. The ClusterSecretStore is one of them, it is a cluster scoped resource that can be referenced by all ExternalSecrets (will explain later) from all namespaces. Use it to offer a central gateway to your secret backend (Secrets Manager in our case).

To create a ClusterSecretStore that links to Secrets Manager, please run this script 03–2-cluster-secret-store.bash. There is nothing much, it will create the ClusterSecretStore resource by using kubectl command. The content of the CRD is shown below or here.

After running the script, we should see the new ClusterSecretStore gcp-secret-store created as shown below. Please note that we don’t need to supply the namespace in the command because it is the cluster scoped resource.

Create secrets in the Secrets Manager

Before pulling secrets from Secrets Manager, we will need to create a Secret (this is not the Kubernetes Secret) in the Secrets Manager first. Run the script 03–3-create-secret-manager.bash to create a Secret.

The script will simply add the JSON content Secret to Secrets Manager. There’re 2 fileds in the JSON and the field values are just the random strings. Once we run the script, we should see new Secret gcp-demo-secret-key in the Secret Manager console as shown below. This Secret will be used by the ExternalSecret CRD for creating the actual Kubernetes Secret later.

It’s not necessary to add only JSON content into the Secret, it can be any formats we want.

Create ExternalSecret CRD

The ExternalSecret is the way to describe how the data should be fetched from the secret backend (Secrets Manager in our case) and how data should be transformed to Kubernetes Secret.

To create the ExternalSecret, please run this script 03–4-external-secret.bash. The script just simply run the kubectl by using the ExternalSecret below.

We may keep the ExternalSecret resources in Git repositories and create them when we create the other Kubernetes workloads (Deployment and StatefulSet for example).

After running the script, we should see the ExternalSecret test-secret-ext created as the result below. Note that we create it in the default namespace.

If everything is OK, we should see the SecretSynced in the STATUS column. If we can see it, we should also able to see the Kubernetes Secret test-secret as shown in the below picture.

Note that the content in the TEST_JSON_SECRETS field is the JSON that we added in the Secret Manager. If we don’t supply the value in the property field in ExternalSecret, the entire content (which is JSON in our case) will be used.

Create a pod and inject Kubernetes Secret into

Now it’s the time to use Kubernetes Secret we just created by the helping of ExternalSecret. We will run this script 04–1-create-pod.bash to create a pod which will mount a Secret test-secret to files inside a pod, under this path /secrets.

After running the script, we should see the pod is running as shown in the below picture.

Now exec into pod to see if secrets are written to files. We should see the result similar to the picture below.

Update Secret in Secret Manager

Now run this script 05–1-update-secret-manager.bash to modify the existing Secret gcp-demo-secret-key in the Secrets Manager and we should see the new secret values similar to the picture below.

It should be no longer than 1 minute (please see field refreshInterval in the ExternalSecret resource) for the External Secret to refresh the new secret values from Secrets Manager and update the related Kubernetes Secret test-secret for us.

Now exec into pod again to see if secret values are changed automatically without pod restarting.