Encrypting Secret data at etcd store on a Minikube K8s Cluster
In this blog let us explore how we can encrypt and store Kubernetes secrets in the etcd key-value store
Before we start, let’s try to understand what is etcd key-value store.
etcd is an open source distributed key-value store used to hold and manage the critical information that distributed systems need to keep running.
It manages the configuration data, state data, and metadata for Kubernetes.
Generally the data that is stored in etcd key-value store is in plain-text format. So anyone who has access to the etcd key-value store can gain access to some sensitive data like Kubernetes secrets.
In this blog we will only focus on securely storing Kubernetes secret objects on the etcd key-value store.
Prerequisites
- Minikube Installation
- Kubectl Installation
- Kubernetes Control Plane running on a Minikube Cluster node
Let us first explore the default behaviour of how Kubernetes secret objects are stored.
Create a Kubernetes Secret object
Let’s first create a Kubernetes generic Secret object named my-secret with data key1=supersecret in the test namespace.
Retrieve the secret we just created in json format,
As you can see from the above output, the data value is stored in a base64 encoded format.
We can decode and view the data stored in the my-secret object using the following command
Now let us check how the my-secret is stored in the etcd key-value store.
View the Secrets in the etcd key-value store
In order to view the secrets values stored in the etcd-key value store, we need to login to the shell of the etcd pod running on the kube-system namespace.
After login to the shell of etcd-multi-node pod (etcd key-value store), we need to execute the following command to fetch the my-secret stored in the etcd key-value store using the etcdctl command.
As you can see from the above screenshot the secret data is stored in plain-text format in the etcd key-value store.
Our goal is to encrypt the secret data stored in the etcd key-value store.
In order to encrypt all the Kubernetes “Secret” resources we need to create a “EncryptionConfiguration” K8s yaml file (encryption-conf.yaml) with Secret as resource to be encrypted and pass it to the kube-apiserver yaml file under container.commands section with “--encryption-provider-config” property.
Before we dive into the steps of it, let us quickly verify the current configurations of the kube-apiserver process running on the Control Plane.
Output of the above command will confirm that the kube-apiserver process is not running with the encryption-provider-config, thus Secrets are not encrypted in the etcd key-value store.
Create a new EncryptionConfiguration Kubernetes resource file
Create a new "EncryptionConfiguration” k8s yaml file
(encryption-conf.yaml) to encrypt all the k8s “secret” resources, this yaml file will be passed as the “--encryption-provider-config” parameter to the kube-apiserver.
Note that the resources only contains the secrets as we are only interested in encrypting the secrets stored in the etcd key-value store.
To generate a new secret for the above config,
Generate a 32-byte random key and base64 encode it.
If you’re on Linux or macOS, run the following command:
In order to point the encryption-conf.yaml file to the
--encryption-provider-config property, we need to mount the file to the kube-apiserver pod with a volumeMount and volume.
Copy the encryption-conf.yaml file in the host machine to the local directory which is mount to the the Minikube node, in my case i have the local $HOME/Minikube/mount directory mount to the /data directory of the minikube node.
Now that the encryption configuration file is available in the Minikube control plane node “/data/encryption-conf.yaml” path.
In order for the encryption-conf.yaml file to be correctly picked up by the
kube-apiserver pod it needs to be placed in the /etc/kubernetes/ path of the Control Plane node and mount as a volume.
So, let’s create a subdirectory encryption in the /etc/kubernetes/ path and copy the encryption configuration file there.
Update the kube-apiserver.yaml
Let’s now edit the “kube-apiserver” definition file and add,
- “--encryption-provider-config” property pointing to the configuration file encryption-conf.yaml from a volumeMount “/etc/kubernetes/enc”
- volume “enc-vol” pointing to the path “/etc/kubernetes/encryption” where we have the encryption configuration file
- volumeMount for the volume “enc-vol” with mountPath “/etc/kubernetes/enc”
Navigate to the “/etc/kubernetes/manifests” location and edit the
“kube-apiserver.yaml” file with the below changes,
Changes to the “kube-apiserver.yaml” file are automatically applied once you save the changes, existing “kube-apiserver” process running on the
“multi-node” cluster node will be destroyed and a new process will be created with the “--encryption-provider-config” parameter we passed.
Let’s view and verify that the “kube-apiserver” process is running with the
“--encryption-provider-config”
This completes all the configuration steps.
Now that the encryption configurations are successfully applied, let us create a new secret with the same data value as my-secret and view how it is stored in the etcd key-value store after the encryption.
Let’s view the new-secret stored in the etcd key-value store.
As you can see the secret data stored in the “etcd” store is completely encrypted.
Now let us try to retrieve this data using the API and check whether we get the correct decrypted data in the base64 encoded format,
As you can see there is no change in the behaviour of data retrieval even after the encryption of etcd key-value store, so all working as expected.
Finally, let us try to view the secrets which are created prior to enabling encryption at the etcd key-value store
As you can see from the above output, enabling encryption does not automatically encrypt existing secrets, it will encrypt any new secrets created after the encryption was enabled.
In order to encrypt all the existing secrets stored on the etcd key-value store, execute the following command,
Above command reads all Secrets and then updates them to apply server side encryption. Since Secrets are encrypted on write, performing an update on a Secret will encrypt that content.
References:
[1] https://etcd.io/
[2] https://minikube.sigs.k8s.io/docs/start/
[3] https://kubernetes.io/docs/tasks/tools/#kubectl
[4] https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/
Hope you enjoyed the blog and got something to take away.
Thank you for Reading!
Cheers!!!
