How Baidu Scales Content Understanding to Trillions of Pages with AI Engineering

Business Background

Baidu indexes massive internet content; to support search it must deeply understand each item, extracting semantics, quality, and safety signals for filtering and semantic indexing. The sheer scale (trillions of items) creates huge computational cost and efficiency challenges.

Key Ideas

Cost Optimization

To meet the massive compute demand Baidu expands its resource pool ("open source") and improves service performance ("throttle"). Elastic scheduling combines idle offline resources with online needs, while model inference is optimized through GPU‑aware code, custom chips, and multi‑process architectures.

Efficiency Optimization

The workflow includes real‑time and offline computation; new features require re‑processing of existing data offline, while newly crawled data is handled in real time. Efficiency is improved by rapid model engineering and faster offline pipelines.

Technical Solutions

Overall Architecture

The core components are the Model Service Platform, Batch Compute Platform, Compute Scheduling System, and Model Service Framework.

Model Service Framework

Algorithms are packaged using a unified Python‑based framework. To overcome Python GIL limitations, Baidu employs a multi‑process, asynchronous coroutine design with separate RPC, DAG, and Model processes, leveraging shared memory and GPU acceleration. Inference is accelerated via dynamic batching, multi‑stream execution, and custom optimizations (Poros, TensorRT, quantization, model compression).

<code>Function classify = {
    def classify(cbytes, ids):
        unique_ids=set(ids)
        classify=int.from_bytes(cbytes, byteorder='little', signed=False)
        while classify != 0:
            tmp = classify & 0xFF
            if tmp in unique_ids:
                return True
            classify = classify >> 8
        return False
}

declare ids = [2, 8];
select * from my_table
convert by json outlet by row filter by function@classify(@cf0:types, @ids);
</code>

Compute Scheduling System

All requests pass through a unified gateway (FeatureGateway) that handles flow control and routing. The SmartScheduler interfaces with multiple internal PaaS resources, automatically deploying operators ("算子") based on resource availability, workload priority, and hardware heterogeneity. Scheduling follows a two‑stage design: traffic scheduling (adjust, sort, assign, bind) and resource scheduling (prepare, hardware‑fit, pre‑assign, group‑assign).

Batch Compute Platform

To address offline bottlenecks, Baidu built an HTAP storage solution separating OLTP and OLAP workloads. The OLAP store uses RocksDB and a custom file system, with columnar layout for hot fields and incremental snapshots for efficient updates. A unified SDK lets users access both Table and OLAP storage.

Summary

The system now supports dozens of search scenarios (image, video, etc.), hundreds of operators, and daily trillion‑scale feature updates. With the rise of large AI models, Baidu plans to further explore model‑driven innovations.