Design and Implementation of Managed (Dedicated) Kubernetes Clusters

The article defines a managed cluster as a setup where a control plane assists users in managing the lifecycle of Kubernetes master components, allowing nodes to be registered to a sub‑cluster via its API server and workloads to be run through the Kubernetes API.

Historically, 360 used Rancher for Kubernetes management; Rancher provides a web UI for deploying and managing containerized applications across multiple hosts.

However, Rancher has several drawbacks: command‑line‑only cluster creation, inflexible custom component definitions requiring full YAML, lack of upgrade control, and inability to configure node‑specific parameters.

Compared with traditional cluster deployment, managed Kubernetes clusters offered by cloud providers provide better scalability, reduced management effort, lower total cost of ownership, enhanced security, and allow developers to focus on application development.

The implemented solution follows a “k8s on k8s” approach, using an existing Kubernetes cluster as the control plane to manage dedicated sub‑clusters, achieving architecture separation, lifecycle management, extensibility, security, and declarative cluster definitions compatible with native Kubernetes APIs.

Cross‑Cluster Interaction Issues

When creating a new sub‑cluster, webhook servers failed to receive requests from the apiserver because the apiserver forwarded requests to a service IP that resolved within the base cluster, never reaching the sub‑cluster pods.

The fix involved enabling the apiserver flag --enable-aggregator-routing=true , allowing requests to be routed to endpoint IPs and requiring webhook pods to use host networking.

Certificate Issuance Challenges

Generating certificates based on pod IPs proved unreliable due to pod recreation and IP changes; the solution was to co‑locate master components in the same pod, sharing the network namespace so they can be accessed via fixed addresses (127.0.0.1/0.0.0.0), and using an LVS IP as a stable whitelist for external access.

Container Runtime Integration

The dedicated cluster supports Docker, containerd, and Kata runtimes; however, mismatched runtimes caused issues with kubelet and kube‑proxy containers, configmap and serviceaccount visibility, and required privileged mode and host networking to resolve.

Adjusting the kubelet specification with opts = append(opts, oci.WithRootfsPropagation("shared")) enabled proper mount propagation, allowing pods to access configmaps, serviceaccounts, and secrets.

Feature Diversity of Dedicated Clusters

Network: classic Cilium‑based VXLAN CNI and Hulk‑VPC‑CNI providing low‑latency, stable networking.

Storage: local LVM‑backed disks and shared PoleFS via CSI.

Logging: integrated ArkIT component.

Monitoring & Alerting: Firehawk‑VMAgent, metrics‑server, node‑exporter, and integrated alerting.

Traffic Control: L4 load balancing with custom load‑balancer and Hulk L4, L7 ingress with domain resolution and ingress‑controller.

Resource Isolation: LXCFS for per‑pod resource view, and offline mixed deployment for high‑density utilization.

Conclusion

The dedicated cluster architecture provides a scalable, secure, and customizable Kubernetes solution with extensive features across networking, storage, logging, monitoring, traffic management, and resource isolation, suitable for users with specific Kubernetes customization needs and future enhancements planned for automated upgrades and declarative node management.