Security Challenges of Kubernetes Secrets and Ant Group’s TEE‑Based End‑to‑End Protection

Kubernetes Secrets store sensitive data such as passwords, certificates, OAuth tokens and SSH keys, making their security critical; any breach can have severe consequences.

InfoQ interviewed Ant Group senior engineer Qin Kailun, who specializes in trusted computing, system security and virtualization, to discuss the security problems of K8s Secrets and the shortcomings of community solutions.

Secrets are objects that centralize the management of sensitive information, avoiding the risk of embedding clear‑text data in pod specifications or container images.

According to Qin, although Secrets reduce accidental leakage, the default implementation still has many security issues: data is stored in etcd only base64‑encoded, the secret and its encryption keys reside in memory in plaintext, and the KMS‑based encryption solution provided by the community leaves keys and plugins vulnerable.

The community‑provided KMS encryption encrypts Secrets at rest but leaves several gaps:

Secret data and encryption keys are kept in memory in clear text, making them easy to steal.

Attackers can impersonate legitimate users to call decryption interfaces and obtain keys.

The KMS plugin’s encrypt/decrypt interfaces cannot prevent impersonation.

Secrets consumed on Nodes remain in plaintext memory, exposing an attack surface.

Ant Group’s approach is to protect the entire lifecycle of a Secret with Trusted Execution Environment (TEE) technology. Their solution includes:

Protecting the KMS plugin that interacts with the API server using TEE, safeguarding root keys and data‑encryption keys while reducing performance overhead.

Securing the KMS provider on the API server with TEE, preventing any secret or key from being exposed in memory and using TEE’s remote attestation to verify callers.

Protecting client‑side tools such as kubectl and kubeconfig with TEE, keeping kubeconfig out of disk and transmitting Secrets through a secure channel directly into TEE.

Protecting the kubelet’s consumption of Secrets on the Node with TEE, eliminating plaintext exposure in memory.

Because no standard or open‑source implementation of a TEE‑based KMS plugin or kubectl exists, Ant Group built its own plugins, offering both standalone and service‑oriented versions to address SGX performance constraints.

They also leveraged the open‑source Occlum LibOS to abstract TEE complexities, making the solution transparent to developers, operators and users.

Qin emphasizes that applying this end‑to‑end TEE protection to Kubernetes Secrets raises the security level of Ant Group’s massive container fleet and is essential for meeting the high data‑security and privacy requirements of financial scenarios.