Understanding Kubernetes Architecture and Custom Controllers: A Practical Guide

This article, originally published by the 360 Qihu operations development team, shares a clear and approachable explanation of Kubernetes (K8s) architecture, its core components, and the standard three‑step process of component registration, resource watching, and action execution, as well as custom compatibility and innovation points introduced for HULK deployments.

The content was first released in the HULK technical discussion series and is authorized for reprint.

Preface

The author has been working with Kubernetes for about five months, during which they read and modified parts of the source code, gaining a basic understanding of its design philosophy and inviting feedback on any gaps.

Author's contributions

Implemented a custom load‑balancer controller (lb‑controller) that integrates with the company’s LVS.

Implemented a custom log‑collection controller (log‑controller) that integrates with the company’s Qbus system.

Modified the RollingUpdate logic to make rolling updates more controllable and smooth for business upgrades.

These three features were developed to adapt the container platform to the company’s existing ecosystem.

Kubernetes Architecture Overview

Kubernetes consists of a Master control plane and multiple Slave nodes. The Master acts as the brain of the system, while the Slave nodes are the work units distributed across the cluster.

The system is composed of many loosely coupled functional components.

etcd Component

etcd serves as the highly available, strongly consistent key‑value store for Kubernetes, providing persistent data storage and service discovery, which is fundamental to many Kubernetes features.

kube‑apiserver Component

kube‑apiserver is the entry point of Kubernetes and the core component besides etcd. It acts as the message bus, wrapping etcd to provide two main functions: a unified query interface for resources and a framework for resource registration and discovery.

All other components (e.g., kube‑controller‑manager, kube‑scheduler, kubelet, kube‑proxy, and the custom controllers) rely on this registration‑and‑discovery mechanism.

Resource Registration and Discovery vs. Service Discovery

While service discovery focuses on locating processes or services (e.g., finding a pod listening on a port), resource discovery aims to detect the existence and state changes of any Kubernetes resource.

kube‑controller‑manager

This component aggregates many resource controllers such as ReplicationController, ReplicaSetController, DeploymentController, ConfigMapController, etc. Controllers watch resources via the list/watch mechanism, react to state changes, and drive the system toward the desired state. For example, the ReplicaSetController watches ReplicaSet and Pod resources, creates Pods to match the desired replica count, and recreates Pods when they fail.

kube‑scheduler

The scheduler watches Pod creation events through the registration‑and‑discovery framework, evaluates node suitability, and binds Pods to appropriate Nodes.

kubelet

Running on each node, kubelet periodically fetches the desired state of Pods/Containers from the API server via the registration‑and‑discovery framework and ensures the node’s actual state matches the desired state.

kube‑proxy

kube‑proxy implements Services by watching Service and Endpoint resources, configuring the appropriate networking rules to route traffic to the correct Pods.

Custom lb‑controller (self‑developed)

Since Kubernetes 1.7 does not support Layer‑4 load balancing, the author built an lb‑controller that listens to Pod add/delete events via the registration‑and‑discovery framework and dynamically updates LVS entries to reflect the current set of Pods.

Custom log‑controller (self‑developed)

The log‑controller integrates Kubernetes with the company’s Qbus log‑collection system (built on Kafka, Logstash, and Qconf). It automatically discovers Pod lifecycle events and configures Logstash accordingly, eliminating manual configuration for rapidly changing Pod resources.

Both custom controllers leverage the client‑go library to interact with the API server, abstracting low‑level details and allowing developers to focus on handling resource state changes.

Conclusion

Through this overview, readers should now have a clearer understanding of Kubernetes' overall architecture and how its modular components interact via the resource registration and discovery framework, encouraging further exploration and practical experimentation.

More Kubernetes knowledge

Available in the 360 Internet Technology Training Camp Phase 9 – “360 Container Technology Decoding and Practice”.