Kubernetes etcd Sensitive Data Encryption Tutorial — Protecting Secrets from Plaintext Storage

When you create a Secret in Kubernetes, its value is displayed in Base64. Therefore, newcomers often mistakenly believe that Secrets are already encrypted.

However, Base64 is encoding, not encryption. Anyone can check the original value with the following command:

echo 'bXlkYXRh' | base64 --decode

A more critical issue is that by default, kube-apiserver does not apply separate encryption at rest when storing Secrets in etcd. If an etcd backup file, data directory, or disk snapshot is leaked, sensitive data can be exposed without going through RBAC.

This article will demonstrate the following steps based on a Kubernetes cluster configured with kubeadm:

Verify Secrets stored in etcd before applying encryption
Generate a 32-byte encryption key
Create EncryptionConfiguration
Link the configuration to the kube-apiserver static Pod
Verify in etcd that new Secrets are encrypted
Rewrite existing Secrets to encrypt all of them
Key rotation and considerations in a production environment

This tutorial is based on the Encrypting Confidential Data at Rest section of the official Kubernetes documentation.

Flow of API server encrypting and storing Secrets in etcd

What you need to understand first: What is being encrypted?

Kubernetes encryption at rest is not a feature for encrypting files mounted on Pods or application databases.

The encryption target is API resources created via the Kubernetes API and persistently stored in etcd. The following resources can typically be specified:

secrets
configmaps
CRD objects, which are custom resources
Other Kubernetes API resources stored in etcd

The data flow is as follows:

kubectl 또는 애플리케이션
        ↓
kube-apiserver
        ↓  EncryptionConfiguration에 따라 암호화
      etcd

When a client queries a Secret via the Kubernetes API, kube-apiserver decrypts the ciphertext from etcd and returns a normal API response. Therefore, users can continue to use kubectl get secret as before.

Encryption at rest is a defense layer against etcd disk or backup leaks. It does not replace the following security controls:

Security Control	Area Protected
Kubernetes RBAC	Controls who can read Secrets via the API
TLS	Protects the transmission path between the API server and etcd
etcd Encryption at Rest	Protects values stored in etcd data and backups
Disk Encryption	Protects entire node disks or volumes
KMS/HSM	Protects encryption keys outside the cluster and manages their lifecycle

Applying one does not eliminate the need for other layers.

Practice Environment and Pre-cautions

This article assumes the following environment:

Kubernetes cluster configured with kubeadm
etcd 3.x
kube-apiserver running as a static Pod on each control plane
sudo access to control plane nodes
kubectl and etcdctl available

Check the cluster version with the following command:

kubectl version

This operation changes core control plane settings. In a production environment, you must back up etcd snapshots and current static Pod manifests, verify the recovery procedure, and then proceed.

sudo cp -a /etc/kubernetes/manifests/kube-apiserver.yaml 
  /root/kube-apiserver.yaml.before-encryption

For a highly available cluster, all control plane nodes must have the same encryption settings and keys. If settings differ between nodes, some API servers might write data that other API servers cannot decrypt.

Step 1: Check if encryption is already applied

First, check if kube-apiserver has the --encryption-provider-config argument.

sudo grep -- '--encryption-provider-config' 
  /etc/kubernetes/manifests/kube-apiserver.yaml

If no results are returned, API server-level encryption at rest is not enabled.

Even if the argument exists, don’t be complacent. If the first provider in EncryptionConfiguration is identity, new data will be stored in plaintext. This is because the provider order determines read and write operations.

When writing: Use the first provider in the list
When reading: Attempt from top to bottom, using the first decryptable provider

During migration, place the encryption provider first and identity for reading old plaintext data later.

Step 2: Create a Secret before applying encryption and check etcd values

Create a test Secret for comparison.

kubectl create secret generic secret-before 
  --namespace default 
  --from-literal=mykey=mydata

When queried via the Kubernetes API, the value appears as Base64.

kubectl get secret secret-before -n default -o yaml

The expected data is as follows:

data:
  mykey: bXlkYXRh

Now, check the original etcd value on the control plane node.

sudo ETCDCTL_API=3 etcdctl 
  --cacert=/etc/kubernetes/pki/etcd/ca.crt 
  --cert=/etc/kubernetes/pki/etcd/server.crt 
  --key=/etc/kubernetes/pki/etcd/server.key 
  get /registry/secrets/default/secret-before | hexdump -C

If encryption is not applied, you should be able to identify the Secret’s structure and the data corresponding to mydata in the output. This is why Base64 display alone cannot protect etcd stored data.

etcdctl connection certificates and endpoints may vary depending on the cluster configuration. If using external etcd, you must specify the actual endpoint and client certificates.

Step 3: Generate a 32-byte encryption key

The official documentation advises creating a 32-byte random key and Base64 encoding it when using a local encryption provider.

KEY=$(head -c 32 /dev/urandom | base64)

It is not recommended to print the key to the screen, copy it to shell history, messengers, or general documents. Anyone with this key can decrypt the encrypted data in etcd.

Create a directory to store the encryption configuration.

sudo install -d -m 700 /etc/kubernetes/enc

Step 4: Create EncryptionConfiguration

In this practice, to understand the storage format and verification flow of the official Kubernetes tutorial, we will encrypt Secrets and ConfigMaps with the aescbc provider.

Caution: The current official Kubernetes documentation classifies aescbc as a weak method due to potential CBC padding oracle attacks and does not recommend it for production environments. The configuration below is for practice to learn the principles of encryption at rest and migration procedures. For new production environments, KMS v2 should be considered first.

sudo tee /etc/kubernetes/enc/enc.yaml >/dev/null <<EOF
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: ${KEY}
      - identity: {}
EOF

unset KEY
sudo chmod 600 /etc/kubernetes/enc/enc.yaml
sudo chown root:root /etc/kubernetes/enc/enc.yaml

The order of providers is important.

providers:
  - aescbc: ...
  - identity: {}

In this configuration, newly created or updated data will be encrypted with aescbc. Existing plaintext data can still be read thanks to the identity provider at the end.

Conversely, if identity is placed first, the API server will continue to store new data in plaintext.

Which to choose: aescbc, aesgcm, secretbox, or KMS?

Kubernetes supports multiple encryption providers, but their operational characteristics differ.

Provider	Features	Operational Considerations
`aescbc`	Simple AES-CBC local symmetric key method	Official documentation classifies it as weak and not recommended for production environments
`aesgcm`	Provides authenticated encryption	Requires key rotation to avoid writing more than ~200,000 items with the same key
`secretbox`	Based on XSalsa20 and Poly1305	Key exists in local configuration file
`kms` v2	Envelope encryption where external KMS manages key encryption keys	Provides strong key isolation, requires separate plugins and availability design

Storing local keys directly in EncryptionConfiguration is better than not using encryption, but the defense is limited if control plane files and etcd data are compromised together. For critical production environments, integration with cloud KMS or HSM should be considered.

Step 5: Link encryption settings to kube-apiserver

Edit the static Pod manifest.

sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml

Add the following argument to kube-apiserver‘s command:

- --encryption-provider-config=/etc/kubernetes/enc/enc.yaml

Add a read-only mount to volumeMounts of the same container.

volumeMounts:
  - name: enc
    mountPath: /etc/kubernetes/enc
    readOnly: true

Add a hostPath volume to Pod spec.volumes.

volumes:
  - name: enc
    hostPath:
      path: /etc/kubernetes/enc
      type: DirectoryOrCreate

When the file is saved, kubelet detects the static Pod change and recreates kube-apiserver.

kubectl -n kube-system get pod 
  -l component=kube-apiserver -w

Check if the API server returns to a healthy state.

kubectl get --raw='/readyz?verbose'

If an error occurs, check kubelet logs and container runtime logs, and recover with the previously backed-up manifest.

sudo journalctl -u kubelet -n 200 --no-pager

Step 6: Verify that new Secrets are stored encrypted

After applying the encryption settings, create a new Secret.

kubectl create secret generic secret-after 
  --namespace default 
  --from-literal=mykey=mydata

Read the value directly from etcd.

sudo ETCDCTL_API=3 etcdctl 
  --cacert=/etc/kubernetes/pki/etcd/ca.crt 
  --cert=/etc/kubernetes/pki/etcd/server.crt 
  --key=/etc/kubernetes/pki/etcd/server.key 
  get /registry/secrets/default/secret-after | hexdump -C

The output should show the following prefix:

k8s:enc:aescbc:v1:key1:

The meaning of each item is as follows:

Value	Meaning
`k8s:enc`	Kubernetes encryption at rest data
`aescbc`	Provider used
`v1`	Storage format version
`key1`	Name of the key used for encryption

Now, verify that decryption via the API is also normal.

kubectl get secret secret-after 
  --namespace default 
  -o jsonpath='{.data.mykey}' | base64 --decode
echo

The following value should be output if normal:

mydata

This means that ciphertext is stored in etcd, but authorized users can still use Secrets normally via the Kubernetes API.

Step 7: Rewrite existing Secrets to encrypt them all

This is the most commonly overlooked part.

Even if encryption settings are enabled, data already stored in etcd is not automatically re-encrypted. Only resources newly created or updated after the settings are applied will be encrypted.

If all existing Secrets are rewritten with the same content, the API server will apply the encryption provider and record them in etcd.

kubectl get secrets --all-namespaces -o json 
  | kubectl replace -f -

If a write conflict occurs, it is safe to run it again. In large clusters, it is better to update Secrets namespace by namespace rather than all at once.

If ConfigMaps were also designated for encryption, they must be rewritten separately.

kubectl get configmaps --all-namespaces -o json 
  | kubectl replace -f -

After migration, check if secret-before, which was created initially, also has the encryption prefix.

sudo ETCDCTL_API=3 etcdctl 
  --cacert=/etc/kubernetes/pki/etcd/ca.crt 
  --cert=/etc/kubernetes/pki/etcd/server.crt 
  --key=/etc/kubernetes/pki/etcd/server.key 
  get /registry/secrets/default/secret-before | hexdump -C

Step 8: Should plaintext fallback be removed?

Once all target data has been sufficiently verified as encrypted, you can remove the identity provider to prevent the API server from reading plaintext stored data.

providers:
  - aescbc:
      keys:
        - name: key1
          secret: <BASE64_ENCODED_32_BYTE_KEY>

However, this step requires caution. If even one unencrypted resource remains, the API server will not be able to read that object.

Therefore, proceed only after fulfilling all the following conditions:

Have all target resources been rewritten?
Are all control planes using the same configuration?
Has the encryption prefix been sampled or fully checked in etcd?
Has the encryption key been securely backed up?
Has the recovery procedure been actually verified?

In the early stages of operation, it is safer to keep identity as a read-only fallback and remove it as a separate change operation after migration verification is complete.

Step 9: Principle of non-disruptive encryption key rotation

Key rotation is not an operation that immediately deletes the old key.

The sequence guided by the official Kubernetes documentation is as follows:

Add the new key as the second key of the existing provider to all control planes.
Apply the settings so that all API servers can read the new key.
Securely back up the new key.
Promote the new key to the first position in the key list.
Apply the settings so that all API servers use the new key for new writes.
Rewrite existing Secrets with kubectl replace.
After verifying that all data is encrypted with the new key, remove the old key.

For example, the configuration in an intermediate step would look like this:

providers:
  - aescbc:
      keys:
        - name: key2
          secret: <NEW_BASE64_KEY>
        - name: key1
          secret: <OLD_BASE64_KEY>
  - identity: {}

The first key is used to encrypt new data, and the subsequent keys are used to decrypt old data. If the old key is deleted before all data is rewritten with the new key, existing Secrets cannot be decrypted.

Kubernetes also supports automatic configuration file reloading with the --encryption-provider-config-automatic-reload=true option. When enabled, the API server periodically checks for file changes, so you don’t have to restart the API server every time you rotate keys. However, the actual application time must be confirmed with relevant metrics.

Order to follow in a highly available control plane

If there are multiple control planes, apply changes sequentially, one node at a time, instead of changing all nodes simultaneously.

Step	Verification Item
1	Securely deploy the same `enc.yaml` to all nodes.
2	Change the first API server’s settings and confirm normal recovery.
3	Change the remaining API servers one by one.
4	Verify that API requests are normal through the load balancer.
5	Check the encryption prefix of new data.
6	Rewrite existing data.

During a rolling change, all API servers must be able to read the current etcd data. Do not promote a new key to the first write key before deploying it to all nodes.

Why KMS is recommended in production environments

The aescbc method used so far is useful for practicing the operation of encryption at rest according to official procedures. However, the current official documentation classifies this method as weak encryption, and the encryption key is also located in /etc/kubernetes/enc/enc.yaml.

If an attacker obtains both etcd data and control plane configuration files, they can simultaneously acquire ciphertext and decryption keys.

Using a KMS provider allows the actual data encryption keys to be protected by an external KMS or HSM. It also makes it easier to leverage the following operational features:

Separation of key access permissions
Centralized key rotation
Key usage audit logs
Utilization of HSM or cloud-managed KMS
Separation of roles between Kubernetes administrators and key administrators

Therefore, this article’s local aescbc configuration should be seen as a starting point for understanding the principles of encryption at rest, and for production clusters requiring a high level of security, it is recommended to consider data encryption using a KMS provider.

Common Mistakes

1. Thinking Base64 of Secret is encryption

Base64 is a representation that anyone can restore. It is separate from etcd encryption at rest.

2. Placing identity as the first provider

Reading is possible, but new data will be stored in plaintext. The write provider is the first item in the list.

3. Applying settings only and not rewriting existing Secrets

Existing data is not automatically converted. A rewrite process like kubectl replace is required.

4. Deploying different keys to each node

In an HA environment, decryption capability may vary per API server, leading to intermittent failures.

5. Deleting old keys without backing them up

Data encrypted with that key will become unrecoverable. If you only have an etcd backup and no key backup, encrypted Secrets cannot be restored.

6. Setting loose permissions on encryption configuration files

The local provider’s key is effectively decryption authority. It should be restricted to root and the API server’s execution principal.

Practice Summary

The key takeaways from this practice are as follows:

Kubernetes Secret’s Base64 is not encryption.
By default, API resources can be written to etcd without encryption at rest.
EncryptionConfiguration‘s first provider encrypts new data.
identity is a fallback for reading existing plaintext data during migration.
Applying settings alone does not encrypt existing data.
You must check for the k8s:enc:aescbc:v1:key1: prefix in etcd.
Keys and etcd backups should be managed together, but access permissions should be separated.
In production environments, external KMS integration provides stronger key protection.

Encryption at rest does not prevent an attacker from gaining Kubernetes API access. Instead, it reduces the risk of sensitive data being immediately exposed if etcd disks, snapshots, and backup files are leaked.

Kubernetes security is complete when RBAC, network protection, audit logs, backup security, and encryption at rest work together.