JDOS Operations Platform: Managing Million‑Scale Container Clusters at JD.com

JD.com operates one of the world’s largest Docker and Kubernetes clusters, achieving a fully containerized environment with millions of containers managed by only two dedicated operators through the JDOS Operations Platform.

The platform was created to address the rapid decline in operational efficiency as cluster size grew, illustrated by regression analysis linking cluster scale (S), container density (s), and servers per operator (m).

The system’s functional tree includes online operations, environment standardization, and intelligent alerting, with a configuration center acting as the “brain” for all cluster metadata.

The operation center handles node actions such as upgrades, log cleaning, password updates, scaling, and deployment, enabling 100% of routine tasks via UI to minimize manual errors.

The first‑generation inspection system built on Ansible proved too slow for tens of thousands of nodes, prompting a redesign using a distributed gossip‑based approach (Serf) to achieve sub‑second convergence even in 100 k‑node clusters.

Serf’s query mechanism executes inspection scripts on target nodes and returns success/failure, reducing a full 10 k‑node inspection from 40 minutes to about 3 minutes.

Intelligent alerting aggregates per‑second container and host metrics via Nodemonitor agents into a TSDB, then applies correlation analysis to converge alerts and predict root causes, dramatically cutting alert noise and improving mean‑time‑to‑resolution.

The overall architecture combines configuration management, inspection, operation, and intelligent alerting to maintain consistency, reduce manual interventions, and enhance operational efficiency in ultra‑large‑scale container environments.