Designing a Cloud‑Native Intelligent Data Architecture for Baidu Search Platform

The Baidu search middle‑platform supports hundreds of retrieval scenarios and billions of content items, but its legacy architecture suffers from manual capacity planning, high cost, and stability issues caused by massive fan‑out and static data management.

To overcome these problems, a cloud‑native intelligent architecture was built, featuring automatic capacity adjustment, on‑demand data storage, and high‑availability design that scales elastically with workload.

The new architecture consists of four core control units: partition controller (defines data partitioning strategies), shard controller (adjusts shard size), replica controller (selects resource packages and replica counts), and routing controller (provides dynamic service discovery and addressing).

Elastic scaling is achieved through horizontal expansion of shard replicas and dynamic shard creation when data volume or traffic grows, reducing capacity‑adjustment latency from weeks to hours.

A resource‑on‑demand mechanism separates hot and cold data, assigning appropriate container specifications to each scenario, which can cut average costs by 30% and up to 80% in typical cases.

Precise fan‑out strategies limit the number of shards involved in a query by aligning data distribution with business attributes (e.g., user ID, shop ID), improving overall availability from 99%⁽¹⁰⁰⁾ to near‑99.9%.

Localized computation aggregates related data into the same shard, eliminating costly distributed joins; in live‑stream e‑commerce search, this reduces average latency by 50%.

Overall, the cloud‑native redesign resolves efficiency and cost issues, while precise fan‑out and localized computation address stability and performance, and future work will automate hot‑cold detection for further optimization.