Design and Evolution of BaikalDB: A Distributed Database for Commercial Product Systems

Commercial advertising platforms demand highly reliable, high‑performance, and cost‑effective data storage that can handle diverse workloads such as OLTP transactions, OLAP analytics, high‑QPS queries, and fuzzy searches. Traditional stacks would require a combination of MySQL, Redis, OLAP warehouses, and search engines.

BaikalDB was created to meet these needs by evolving from a single‑node MySQL deployment to a multi‑region, multi‑replica distributed system. The design journey includes three major stages: early MySQL sharding, a heterogeneous storage cluster with real‑time MySQL‑to‑memory synchronization, and finally a cloud‑native, MySQL‑compatible database built on RocksDB, Raft, and Baidu’s brpc framework.

The system’s architecture consists of three core components: BaikalStore for data storage (organized into Regions with Raft‑based three‑replica groups), BaikalMeta for metadata and cluster management, and BaikalDB for stateless SQL parsing and query planning. It supports flexible cloud deployment, one‑stop storage‑compute capabilities, and MySQL protocol compatibility.

In the storage layer, data is mapped from relational tables to a key‑value model using RocksDB. Primary keys, local and global secondary indexes, and full‑text indexes are encoded as KV pairs, with Region‑level range sharding to balance load. Raft ensures strong consistency across replicas, while two‑phase commit and Percolator‑style transaction coordination provide distributed transaction support.

The compute layer employs a volcano‑style operator model, combining rule‑based and cost‑based optimizers to select the best execution plan. Index selection, join processing, and filter push‑down are handled to minimize data movement, and the system can execute distributed computation on storage nodes, though final aggregation occurs on a single BaikalDB node.

Scheduling is managed by BaikalMeta, which collects heartbeats from BaikalStore nodes and makes leader‑balance and peer‑balance decisions to evenly distribute load and ensure fault tolerance. Regions are split automatically when they exceed size thresholds, and the scheduler can relocate leaders or peers to maintain performance and availability.

Since its 2018 launch, BaikalDB has grown to over 1.5 K tables, 600 TB of data, and more than 1.7 K storage nodes, consolidating the previously fragmented storage stack into a unified platform that continues to evolve with new business scenarios.