etcd Encryption at Rest: Configuration, Key Rotation, and Performance Impact

Problem

Kubernetes Secrets are stored in etcd as base64-encoded plaintext. Base64 is an encoding, not encryption. Anyone with direct access to the etcd data directory, an etcd backup, or the etcd API can read every secret in the cluster: database passwords, API keys, TLS certificates, and service account tokens.

This is not a theoretical risk. etcd backups are stored on disk or in object storage. If those backups are not encrypted at the storage level, every secret is exposed. If an attacker gains access to the control plane node filesystem, they can read etcd data files directly. If etcd is exposed without TLS client authentication (a common misconfiguration in self-managed clusters), secrets are accessible over the network.

The Kubernetes EncryptionConfiguration API solves this by encrypting secrets before they are written to etcd. But the implementation has real operational costs:

• Key management is your responsibility. You generate the encryption key, store it on the control plane node, and rotate it manually. If you lose the key, you lose access to all encrypted secrets.
• Key rotation requires control plane coordination. Rotating keys means updating the encryption config, restarting the API server, and re-encrypting all existing secrets. On multi-control-plane clusters, this must be coordinated across all nodes.
• Performance impact is measurable. Encryption adds 1-5% latency to secret read/write operations. For clusters with high secret churn (frequent deployments, many short-lived tokens), this is noticeable.
• Managed providers handle this by default. If you use a managed Kubernetes service, etcd encryption is typically enabled and managed for you. This article is primarily for self-managed clusters.

Target systems: Self-managed Kubernetes 1.29+ clusters (kubeadm, k3s, RKE2). Managed providers (EKS, GKE, AKS, Civo, DigitalOcean) handle etcd encryption automatically.

Threat Model

• Adversary: Attacker with access to etcd data at rest: control plane node filesystem, etcd backups in object storage, or etcd snapshots on a compromised CI/CD system.
• Access level: File system read access to etcd data directory (/var/lib/etcd/) or access to etcd backup files.
• Objective: Read Kubernetes Secrets (database credentials, API keys, TLS private keys, OAuth tokens) from the stored etcd data.
• Blast radius: Without encryption at rest, all secrets in the cluster are exposed in plaintext. With encryption, the attacker gets ciphertext that is unusable without the encryption key. The key itself must be stored separately and protected.

Configuration

Step 1: Generate an Encryption Key

# Generate a 32-byte (256-bit) random key, base64-encoded
ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)
echo "Generated key: $ENCRYPTION_KEY"

# Store the key securely (outside the cluster)
# This key is the master secret. If you lose it, encrypted data is unrecoverable.

Step 2: Create the EncryptionConfiguration

# /etc/kubernetes/encryption-config.yaml
# This file must exist on every control plane node.
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps  # Optional: encrypt configmaps too
    providers:
      # First provider is used for WRITING new data
      - aescbc:
          keys:
            - name: key-2026-04-22
              secret: <base64-encoded-32-byte-key>
      # Identity provider allows READING unencrypted data
      # (existing secrets written before encryption was enabled)
      - identity: {}

Provider comparison:

Step 3: Configure the API Server

For kubeadm clusters:

# /etc/kubernetes/manifests/kube-apiserver.yaml
# Add the encryption-provider-config flag
apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
    - name: kube-apiserver
      command:
        - kube-apiserver
        - --encryption-provider-config=/etc/kubernetes/encryption-config.yaml
        # ... other existing flags ...
      volumeMounts:
        - name: encryption-config
          mountPath: /etc/kubernetes/encryption-config.yaml
          readOnly: true
  volumes:
    - name: encryption-config
      hostPath:
        path: /etc/kubernetes/encryption-config.yaml
        type: File

# For kubeadm, edit the static pod manifest directly:
sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml

# Add to the command section:
# --encryption-provider-config=/etc/kubernetes/encryption-config.yaml

# Add volume and volumeMount for the config file.
# The kubelet will automatically restart the API server.

For k3s:

# Create the encryption config
sudo mkdir -p /var/lib/rancher/k3s/server/
sudo cp encryption-config.yaml /var/lib/rancher/k3s/server/encryption-config.yaml

# Add to k3s server configuration
# /etc/rancher/k3s/config.yaml
# kube-apiserver-arg:
#   - "encryption-provider-config=/var/lib/rancher/k3s/server/encryption-config.yaml"

sudo systemctl restart k3s

Step 4: Encrypt Existing Secrets

Enabling encryption only affects newly written secrets. Existing secrets remain unencrypted until they are re-written.

# Re-encrypt all secrets in the cluster:
kubectl get secrets --all-namespaces -o json | \
  kubectl replace -f -

# Verify the operation completed:
echo "Re-encrypted $(kubectl get secrets -A --no-headers | wc -l) secrets"

Step 5: Verify Encryption Is Active

# Read a secret directly from etcd (requires etcd client access)
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/my-secret

# Expected output: binary/encrypted data prefixed with "k8s:enc:aescbc:v1:key-2026-04-22"
# If you see plain base64 JSON, encryption is NOT active.

# Quick verification script:
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/my-secret | \
  hexdump -C | head -5

# Encrypted: you will see binary data with "k8s:enc:aescbc" prefix
# NOT encrypted: you will see readable JSON with "apiVersion" and "data"

Step 6: Key Rotation

Key rotation requires updating the encryption config, restarting the API server, and re-encrypting all secrets with the new key.

# 1. Generate a new key
NEW_KEY=$(head -c 32 /dev/urandom | base64)

# 2. Update encryption-config.yaml
# Add the new key as the FIRST key (used for new writes)
# Keep the old key as the SECOND key (used to decrypt old data)
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key-2026-07-22  # New key (first = write key)
              secret: <new-base64-key>
            - name: key-2026-04-22  # Old key (decrypt only)
              secret: <old-base64-key>
      - identity: {}

# 3. Restart the API server on each control plane node
# For kubeadm: the kubelet watches the static pod manifest
sudo cp encryption-config.yaml /etc/kubernetes/encryption-config.yaml
# Wait for API server to restart (10-30 seconds)
kubectl get nodes  # Verify API server is responding

# 4. Re-encrypt all secrets with the new key
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

# 5. After confirming all secrets are re-encrypted, remove the old key
# Update encryption-config.yaml to remove the old key
# Restart the API server again

For multi-control-plane clusters: Update the encryption config on ALL control plane nodes before re-encrypting secrets. If node A has the new key but node B does not, reads routed to node B will fail for secrets encrypted with the new key.

Step 7: KMS Provider (Vault Integration)

For production clusters, use a KMS provider so the encryption key never exists on disk.

# /etc/kubernetes/encryption-config.yaml (KMS v2)
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          apiVersion: v2
          name: vault-kms
          endpoint: unix:///var/run/kms-plugin/kms.sock
          timeout: 3s
      - identity: {}

The KMS plugin runs as a sidecar or DaemonSet that communicates with Vault over a Unix socket. The API server sends data to the plugin for encryption/decryption; the plugin uses the Vault transit engine, and the actual encryption key never leaves Vault.

Expected Behaviour

After enabling etcd encryption at rest:

• New secrets are encrypted before being written to etcd
• Existing secrets are encrypted after the re-encryption step
• etcdctl get /registry/secrets/... returns binary data with a k8s:enc:aescbc prefix
• kubectl get secret still works normally (the API server decrypts on read)
• Secret creation and retrieval latency increases by 1-5% (measurable, rarely noticeable)
• etcd backups contain encrypted secrets (unreadable without the encryption key)
• Key rotation completes without downtime (rolling API server restarts)

Trade-offs

Failure Modes

When to Consider a Managed Alternative

Transition point: Managing encryption keys for a self-managed Kubernetes cluster is a permanent operational responsibility. Key rotation must happen on a schedule (quarterly is typical). Key backup must be tested. Multi-control-plane coordination adds complexity to every rotation. If you are running Kubernetes primarily for application workloads (not as a platform product), the key management burden is pure overhead.

Recommended providers:

• Civo (#22) and DigitalOcean (#21): Managed Kubernetes services that encrypt etcd at rest by default. You never see or manage the encryption key. Key rotation is handled automatically. This eliminates the entire operational burden described in this article.
• HashiCorp Vault (#65): If you must run self-managed Kubernetes, use Vault as a KMS provider. The encryption key never exists on the control plane node filesystem. Vault handles key rotation, access logging, and key lifecycle management. The operational burden shifts from “manage a key file on disk” to “manage Vault,” which you likely already do for application secrets.

What you still control: The decision of which resources to encrypt (secrets only, or configmaps too), the rotation schedule, and the KMS provider selection. Managed providers make the encryption invisible, which is the correct answer for most teams.

Premium content pack: Ansible playbook for automated etcd encryption setup on kubeadm clusters, including key generation, config distribution to all control plane nodes, API server restart coordination, and verification. Includes a CronJob-based key rotation automation with Slack notification.