Baidu Search Stability Issue Analysis: Automated Fault Detection and Resolution Techniques

Baidu Search, one of the largest and most critical online services, must maintain extreme availability, prompting the development of sophisticated stability‑governance techniques to quickly identify and resolve faults.

The authors identify eight core challenges: fast log retrieval, balancing real‑time analysis with accuracy, comprehensive fault description, effective feature extraction, query‑scene reconstruction, deep fault feature mining, cascade‑failure detection, and handling unknown faults.

To address rapid log access, an index‑mirroring system stores query‑ID‑based log locations in an in‑memory side‑index, enabling O(1) retrieval of relevant logs.

Streaming analysis is introduced to trigger automatic, incremental fault analysis as soon as a rejection signal or new log arrives, ensuring sub‑second response.

A complete label‑set is built by enumerating all possible module‑level failure reasons, providing exhaustive coverage for fault attribution.

Feature engineering is realized via a rule‑extraction engine that converts raw logs into binary or numeric features, which are then vectorized for matching against failure reasons.

Single‑query scene reconstruction uses span‑id tracing across modules to rebuild the full dispatch tree, allowing precise pinpointing of the failure path.

An intelligent ranking algorithm leverages entropy‑based scoring to surface the most clustered dimensions of failure features, guiding root‑cause identification.

A timeline analysis visualizes per‑second rejection counts and trends, helping operators see the evolution of incidents.

Chaos engineering techniques inject controlled faults into the live system, generating labeled samples for unknown‑fault detection and improving predictive capabilities.

Additional mechanisms include long‑tail batch analysis that traverses full tracing data to locate latency outliers, and full‑process abnormal‑state tracking that correlates cache hits/misses with query failures.

Collectively, these eight technologies achieve up to 99% fault‑analysis accuracy, delivering module‑level rejection reasons within seconds and scaling to massive query volumes.