Tera: Baidu’s Large‑Scale Distributed Table System and Its Role in Real‑Time Search

Information technology is advancing rapidly, and the explosion of network data challenges the real‑time capabilities of search engines. Baidu processes trillions of link analyses and hundreds of billions of web resource crawls daily, relying on its core database Tera to handle trillion‑scale real‑time data.

At a Baidu technical salon, Baidu’s web search infrastructure managers introduced Tera’s application in the search engine, describing how it supports real‑time link storage, index filtering, and user‑behavior analysis. Tera is a large‑scale distributed table storage system designed for high performance and scalability, originally built to manage trillion‑level hyperlinks and web pages.

In the link‑storage stage, Baidu’s Spider 3.0 system uses Tera as its real‑time backbone, turning batch MapReduce processing into real‑time operations that achieve billions of random reads/writes per second and process trillions of link updates daily.

For index filtering, Tera stores all intermediate data—including web, deduplication, and result tables—and enables streaming computations that merge page features, calculate page value, deduplicate, and sort indexes in real time, reducing processing latency from days to minutes or seconds.

In user‑behavior analysis, Tera’s real‑time data streams lower latency to seconds, allowing instantaneous detection of spikes, intent analysis, and product iteration evaluation, thereby improving timeliness and user experience.

The design of Tera follows a BigTable‑style key‑value model with global ordering. Its architecture consists of TabletServers, a Master, and a ClientSDK. TabletServers manage data partitions (Tablets) and LocalityGroups, which can be stored on different media for performance optimization.

High availability is achieved through ZooKeeper‑based TabletServer registration and automatic tablet migration upon failures, as well as load‑balancing strategies that consider SSD capacity, read/write load, and pending request volumes.

Performance optimizations include making the system friendly to distributed file systems (WAL persistence and sync), using SSDs as cache while persisting most data on SATA, and designing all potentially blocking operations to be asynchronous, complemented by client‑side tablet location caching and an enhanced Bloom filter for ~20% read performance gains.

Overall, Tera transforms Baidu’s search engine from batch‑oriented processing to incremental, real‑time computation, dramatically improving the engine’s ability to handle massive, time‑critical data workloads.