High‑Availability Practices of ClickHouse in JD.com: Architecture, Deployment, and Operations
The article details JD.com’s large‑scale OLAP strategy using ClickHouse as the primary engine and Doris as a secondary engine, covering application scenarios, component selection criteria, cluster deployment models, high‑availability architecture, fault‑handling procedures, performance tuning, and future cloud‑native plans.
JD.com operates an OLAP platform serving transaction, traffic, user analysis, and large‑event dashboards, handling billions of queries and trillions of rows daily across roughly 3,000 servers.
Key challenges include multi‑table joins, massive data scans, real‑time low‑latency requirements, and the need for flexible, scalable analysis across diverse business domains.
Component selection focuses on massive data volume support, adaptability to varied workloads, flexibility for dynamic metric adjustments, and minute‑level data freshness.
The chosen solution combines ClickHouse for performance and scalability with Doris for ease of maintenance, supplemented by a custom management layer to lower operational overhead.
Cluster deployment adopts a small‑cluster multi‑tenant model, offering three modes: large monolithic clusters, independent per‑business clusters, and medium‑sized shared clusters, each with trade‑offs in isolation and resource utilization.
High‑availability is achieved through physical isolation, multi‑tenant quotas, workload‑specific isolation (analysis vs. reporting, high‑concurrency vs. low‑latency), and a three‑layer architecture comprising DNS resolution, CHProxy for request routing, and ClickHouse nodes, supported by ZooKeeper for metadata coordination.
Node lifecycle management (offline, online, replacement) is automated via configuration distribution and ZooKeeper metadata synchronization, ensuring minimal disruption.
A dual‑active cluster design enables rapid failover between data centers, with data replicated via Spark (offline) and Flink (real‑time) to maintain eventual consistency.
Extensive parameter tuning across Linux, ZooKeeper, and ClickHouse (e.g., memory limits, inode counts, max_memory_usage) stabilizes performance under heavy load.
Common operational issues—system OOM, partition explosion, metadata inconsistencies, query skew, and ClickHouse join inefficiencies—are addressed through quotas, query rewriting, materialized views, and careful schema design.
Future work includes a Raft‑based replacement for ZooKeeper to improve metadata handling, enhanced OLAP control panels for self‑service provisioning and automated diagnostics, and a cloud‑native, containerized OLAP architecture leveraging external storage and elastic scaling.
DataFunTalk
Dedicated to sharing and discussing big data and AI technology applications, aiming to empower a million data scientists. Regularly hosts live tech talks and curates articles on big data, recommendation/search algorithms, advertising algorithms, NLP, intelligent risk control, autonomous driving, and machine learning/deep learning.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.