How Baidu’s Staged Release and Intelligent Health Checks Prevent Deployment Failures

Overview

Developers constantly deliver new features, fix bugs, and improve performance, but each change carries risk. More than half of major outages stem from changes, making it essential to monitor service health during releases and stop faulty deployments early.

Staged Release for Controllable Impact

Baidu uses a staged release mechanism that splits deployment into multiple phases, applying changes to a subset of machines at each step and checking service health before proceeding. If health degrades, the release can be halted or rolled back, limiting the fault to only the machines that have received the change.

The optimal practice consists of five stages: sandbox, a few machines in a single data center, all machines in that data center, a few machines in other data centers, and finally all machines in all data centers. This balances risk and efficiency.

Why Health Checks Matter

Health‑check monitoring between adjacent stages is the core of staged release. Without effective checks, faults discovered late can affect the entire fleet.

Manual Checks Are Too Slow

Operators manually inspect hundreds of metrics (CPU, request volume, etc.) after each stage, but they only have about ten minutes total, which translates to roughly 0.5 seconds per metric—far too fast for human analysis.

Rule‑Based Automated Checks

To automate, engineers define threshold rules for each metric. After a release, a script compares current metric values against these thresholds. If any metric falls outside its range, the release is halted.

However, this approach faces two major challenges: selecting appropriate thresholds and updating them as traffic patterns evolve.

Intelligent Checks Eliminate Manual Thresholds

Intelligent checking analyzes metric spikes and determines whether they are caused by the change or by external factors such as traffic growth or process restarts. It uses two perspectives:

Time‑factor influence: compare the experimental group (machines with the change) against a control group (machines without the change) to filter out global traffic effects.

Restart influence: model metric behavior after process restarts using historical change data.

If a metric’s abnormal change cannot be explained by either factor, it is flagged as an anomaly and the release is stopped.

Conclusion

By combining staged releases with automated health‑check scripts and AI‑driven anomaly detection, Baidu reduces the risk and impact of change‑induced failures without requiring manual threshold tuning, offering a scalable and efficient solution for modern operations.