Inside KubeWharf: ByteDance’s Open‑Source Cloud‑Native Suite for Scalable Kubernetes

Open Source Background

In 2016 ByteDance adopted Kubernetes and began large‑scale containerization, reaching tens of thousands of nodes by 2018. Today more than 95% of its applications are cloud‑native, and the company is open‑sourcing its large‑scale cloud‑native tools and best practices through the KubeWharf project.

KubeBrain

KubeBrain is a high‑performance metadata storage system that replaces etcd for Kubernetes. It improves read/write performance, reduces OOM risk, and offers stateless operation, extensibility, high availability, compatibility with etcd APIs, and horizontal scalability.

High Performance : Optimized read/write logic and storage engine give clear advantages over etcd and lower OOM risk.

Stateless : Acts as a storage interface for the API server while actual data resides in the underlying engine; data is cached in the master node’s memory.

Extensibility : Abstracts a key‑value DB interface, allowing any KV store with required features to be used.

High Availability : Master‑slave architecture with leader election provides automatic failover.

Compatibility : Supports the etcd API for seamless Kubernetes integration.

Horizontal Scaling : Adds read‑only follower nodes for higher concurrency and can expand storage nodes at the engine layer.

KubeGateway

KubeGateway is a seven‑layer HTTP/2 load‑balancing proxy designed for kube‑apiserver traffic, offering flexible routing, request governance, connection reuse, hot configuration updates, and gateway capabilities such as rate limiting and circuit breaking.

Load Balancing : Performs request‑level balancing across multiple apiserver instances, enabling true horizontal scaling.

Request Governance : Routes based on verb, apiGroup, resource, user, service account, etc., supporting scenarios like separating pod and node traffic or gray‑release upgrades.

Connection Reuse : Reuses HTTP/2 connections to reduce TCP connections per apiserver instance by an order of magnitude.

Hot Configuration Updates : Routing and other settings take effect instantly without restarting services.

Gateway Features : Provides dynamic service discovery, throttling, degradation, circuit breaking, caching, and blacklist/whitelist controls.

KubeZoo

KubeZoo is a lightweight multi‑tenant solution that virtualizes multiple control planes on a single physical Kubernetes cluster, delivering low resource consumption, high control‑plane isolation, reduced operational cost, and rapid tenant provisioning.

Low Resource Consumption : Uses a single gateway instead of separate control‑plane clusters for each tenant.

High Control‑Plane Isolation : Each tenant sees a complete Kubernetes view, with access to both namespace‑scoped and cluster‑scoped resources.

Low Operational Cost : Fewer control‑plane clusters dramatically cut maintenance and upgrade effort.

High Efficiency : Tenants can be created in seconds by creating a Tenant object, bypassing hardware allocation and control‑plane initialization.

Project RoadMap

The first three projects—KubeBrain, KubeGateway, and KubeZoo—are now open‑source. Future plans include continuous iteration based on internal and external feedback, enhancing stability, performance, and usability; expanding gateway capabilities and exploring multi‑cluster federation; and integrating technologies like Virtual Kubelet and Kata to shape serverless Kubernetes offerings. Additional internal projects such as a high‑performance distributed scheduler and mixed‑workload control system will also be open‑sourced, forming the basis of a Kubernetes distribution tailored for large‑scale, multi‑tenant, mixed‑workload scenarios.

Scan the QR code to join the KubeWharf project chat group.