How to Secure Multi‑Tenant Kubernetes Clusters: Best Practices and Architecture

Solving security isolation for multi‑tenant clusters is crucial for enterprises moving to the cloud. This article discusses the concept of Kubernetes multi‑tenant clusters, common usage patterns, enterprise shared‑cluster scenarios, and existing Kubernetes security management features.

What is a Multi‑Tenant Cluster

First, a “tenant” includes not only cluster users but also workloads that consume compute, network, storage, and other resources. In a multi‑tenant cluster, different tenants are isolated so that a malicious tenant cannot attack others, and shared resources are allocated fairly.

Based on the security level of isolation, clusters can be classified as Soft Multi‑tenancy and Hard Multi‑tenancy. Soft isolation is suitable for internal enterprise clusters where malicious tenants are unlikely; it mainly protects business between internal teams. Hard isolation is required for service providers offering external services, where tenants may have unknown security backgrounds and strict isolation is needed.

Multi‑Tenant Use Cases

Below are two enterprise multi‑tenant solutions with different isolation requirements.

Enterprise Shared‑Cluster Multi‑Tenant

In this scenario all cluster users are from the same enterprise, which is a common usage pattern for many Kubernetes clusters. Because user identities are controllable, security risks are relatively low. Enterprises can use namespaces to logically isolate resources for different departments or teams. The following roles are defined:

Cluster Administrator

Has cluster‑wide management capabilities such as scaling and node addition.

Creates and assigns namespaces for tenant administrators.

Manages policies such as RAM, RBAC, NetworkPolicy, and quotas.

Tenant Administrator

At least read‑only access to cluster RAM.

Manages RBAC configuration for users within the tenant.

Tenant User

Uses Kubernetes resources within the allowed namespace scope.

In addition to role‑based access control, network isolation between namespaces must be enforced so that only whitelisted cross‑tenant requests are allowed.

SaaS and KaaS Service Models

In a SaaS multi‑tenant scenario, tenants are the SaaS platform and the service‑application instances running on the Kubernetes control plane. Each tenant’s workloads run in separate namespaces, and end users interact with the SaaS console rather than the Kubernetes control plane directly.

In a KaaS scenario, the business platform’s services are exposed directly to tenants via the Kubernetes API or other extended APIs. Isolation is achieved through namespaces, network policies, and resource quotas, but because end users may come from untrusted zones, stronger security measures such as kernel‑level isolation are required.

Implementing Multi‑Tenant Architecture

When planning and deploying a multi‑tenant cluster, start with a resource isolation model that separates the cluster itself, namespaces, nodes, Pods, and containers. Controlling the resource domain each tenant can access reduces security risks. At the scheduling layer, ensure that containers handling sensitive data run on dedicated nodes, and use runtime scheduling policies to limit cross‑tenant attacks.

Although native Kubernetes security and scheduling features are insufficient for complete isolation, namespace isolation combined with RBAC, PodSecurityPolicy, NetworkPolicy, and other policy models can provide reliable security boundaries.

The following sections focus on native Kubernetes security features for multi‑tenant practice.

Access Control

NetworkPolicy

NetworkPolicy controls network traffic between Pods of different tenants and uses whitelist rules for cross‑tenant access.

PodSecurityPolicy

PodSecurityPolicy (PSP) is a cluster‑level resource that validates Pod specifications during admission, checking for host network, host filesystem, privileged containers, and other security contexts. It can restrict privileged container usage and enforce UID/GID, kernel capabilities, and other settings.

OPA

Open Policy Agent (OPA) is a powerful policy engine that decouples policy decisions from the platform. When namespace‑level RBAC is insufficient, OPA can provide fine‑grained object‑level access control, support multi‑layer NetworkPolicy definitions, and enable label‑ or annotation‑based cross‑namespace policies.

Resource Scheduling

ResourceQuota and LimitRange

In multi‑tenant environments, teams share cluster resources, which can lead to contention. ResourceQuota limits total resource requests per namespace, while LimitRange sets default request and limit values for Pods. Quotas can also be applied to storage and object counts.

Pod Priority and Preemption

Since Kubernetes 1.14, Pod priority and preemption allow higher‑priority Pods to evict lower‑priority ones when resources are scarce, ensuring critical workloads remain available. Combining priority with ResourceQuota can protect important tenant applications.

Dedicated Nodes

Note: Malicious tenants may add matching tolerations to bypass node taints. The following applies only to trusted tenant clusters.

By adding taints to specific nodes, administrators can reserve those nodes for particular tenants (e.g., GPU nodes). Pods must have matching tolerations to be scheduled on tainted nodes, but malicious tenants could add the same tolerations, so additional controls are needed.

Protect Sensitive Information – Secret Encryption

All tenants share the same etcd store. Encrypting Kubernetes secrets protects sensitive data from accidental exposure due to misconfigured access controls.

Summary

When deploying a multi‑tenant architecture, first identify the trust level of tenants and the required isolation degree. To meet basic isolation requirements, it is recommended to:

Enable Kubernetes default security settings.

Enable RBAC and disable anonymous access.

Enable secret encryption.

Follow the CIS Kubernetes Benchmark.

Enable admission controllers such as NodeRestriction, AlwaysPullImages, and PSP.

Use PSP to control privileged Pod deployment and enforce security contexts.

Configure NetworkPolicy.

Enable Seccomp, AppArmor, and SELinux for the Docker runtime.

Isolate monitoring, logging, and other services per tenant.

For SaaS, KaaS, or untrusted tenant scenarios, stronger isolation measures include:

Use OPA DENG dynamic policy engine for fine‑grained network or object‑level access control.

Deploy secure containers with kernel‑level isolation.

Implement comprehensive multi‑tenant isolation for monitoring, logging, storage, and other services.

Source: https://medium.com/@Alibaba_Cloud/practices-of-kubernetes-multi-tenant-clusters-bde2b33fbd91