Baidu's High-Performance KV Storage System Practice: Cloud-Native Optimization and Software-Hardware Co-Design

This article introduces Baidu's practice of building high-performance KV storage systems using cloud-native, full-user-state, and software-hardware协同 technologies. Since 2016, Baidu's "Search + Information Feed" dual-engine驱动构建内容生态, with KV storage serving as the widely-used online storage service in search and recommendation middleware, facing dual tests of storage scale and access traffic.

By early 2018, Baidu had deployed over 10,000 servers for various KV storage services, with data scale exceeding 100PB, handling nearly 100 billion daily access requests. The challenges included: 1) Performance challenges - balancing read/write performance, latency, and space utilization in engine design; 2) Cloud-native challenges - containerization transformation requiring fine-grained control of memory and storage I/O, and dynamic management requiring fault tolerance and elastic scaling; 3) Meeting specialized business requirements while maintaining common capabilities.

The solution, UNDB (United NoSQL Database), implements: Key-Value separation engine reducing write amplification to <1.5x with space amplification <1.6x; OpenChannel SSD-based software-hardware协同 engine achieving >30% performance improvement with overall amplification <1.1x; Cloud-native architecture with Operator-based global data scheduling; Multi-model storage architecture with pluggable layers for interfaces, data models, data synchronization, and engines.

Results: The system covers Baidu's main information feed and search scenarios with hundreds of thousands of instances, handling over 1 trillion daily access requests, reducing costs by nearly 50%.