Security Risks of Exposing Private Keys in Istio Service Mesh and Mitigation Approaches

The authors, Zhang Huilong and Yang Ailin, are engineers at Intel working on cloud‑native service mesh technologies; Zhang contributes to the Istio and Linkerd communities, while Yang focuses on cloud orchestration for Intel's cloud‑native projects.

During the investigation of Istio Ingress Gateway use cases, a security vulnerability was discovered: when a workload’s private key is uploaded to the mesh via a Kubernetes secret, the key is effectively stored in plaintext (base64‑encoded) and can be accessed from memory or the secret itself.

Two methods exist for generating workload private keys in Istio: (1) generating mTLS keys inside the mesh, and (2) uploading the key through an Ingress gateway. The article concentrates on the second method, highlighting three main risks: the key is nearly plaintext, it resides in memory, and it is uploaded externally.

Using openssl rsa -in private_file_name -text -noout , the base64‑encoded key can be decoded, revealing the raw private key. The article demonstrates this with several figures showing the decoded output.

Further, the private key may be stored in the sidecar’s memory (Envoy). By attaching GDB to the Envoy process (e.g., gdb followed by attach <Process ID> ), one can inspect the memory mappings, locate the heap region, and search for the PEM markers “BEGIN PRIVATE KEY” and “END PRIVATE KEY” using the find command with their hexadecimal representations. The author provides example address ranges and shows how to identify potential key locations.

Although GDB is used in the demonstration, other memory‑dump tools can achieve the same result, confirming that storing private keys in memory is insecure.

To protect high‑security workloads, the article recommends end‑to‑end protection of private keys—from generation to storage and usage. It points to an Intel SGX‑based solution contributed to the Istio community, as described in the referenced CNCF blog post, and mentions the HSM SDS Server project for further details.

References: [1] https://www.cncf.io/blog/2021/12/06/hardware-based-security-for-service-mesh-keys/ [2] https://github.com/istio-ecosystem/hsm-sds-server