Stability Governance in Tencent Search: Architecture, Incident Management, and Automation

This article presents the comprehensive stability governance practice of Tencent Search, focusing on improving system availability, incident detection, and rapid response.

1. Availability Architecture – The system adopts a multi‑layered architecture that includes redundancy (multi‑region, multi‑instance), pre‑emptive interception, automated defense, and risk mitigation. Redundant deployment across data centers and instances ensures that failures in a single node do not cascade.

2. Disaster Recovery (Resilience) – Various disaster‑recovery mechanisms are described, such as DNS‑level traffic switching, Nginx‑level flow redirection, and custom routing rules in the internal “North Star” platform. A dedicated SearchGuard cache service is introduced to provide cold‑standby data when the primary path is unavailable.

3. Detection (Monitoring) – The detection framework combines black‑box, business, functional, statistical, engineering, and infrastructure metrics. Key indicators like MTTD (Mean Time To Detect) and KPI phone alerts are used to achieve minute‑level detection. Monitoring is kept minimal yet precise, with alert routing to enterprise WeChat groups.

4. Emergency (Rapid Response) – An emergency workflow accelerates five steps: fast notification, quick team entry, rapid loss‑cutting, swift decision making, and fast recovery. The process is supported by a unified platform that provides one‑click actions, experiment suspension, and automated rollback.

5. Interception (Pre‑Release Guard) – A multi‑stage gating system (pre‑release, CD tiered rollout, sandbox) intercepts risky changes early. Detailed checklists and automated validation pipelines reduce the chance of incidents reaching production.

6. Automation (Zero‑Effort Governance) – Automation is applied to monitoring, alert handling, deployment pipelines, and good‑case testing. The article showcases a command‑protocol snippet used by the degradation platform:

// 指令协议
1 key = value形式的string，其中key为xxDegrade；value为各指令的组合字符串；
2 一个指令最多有三级参数
3 各指令之间用英文|分割
4 指令的二级参数用英文:指定
5 若指令的二级参数有多个，之间用&分隔
6 若指令有三级参数，之间用#连接

// 指令名称
$支路$服务名称$降级内容，sogou侧用sg开头，中台侧用kd开头；

// 实例1（一级指令和二级指令组合）
xxDegrade = sgZhiling1|sgXX:1&2&3|kdXX:15

// 实例2（三级指令）
xxDegrade = sgXX:XX#1000&XX#300

7. Governance Model – The article contrasts top‑down governance with a “bottom‑up” model where each team owns its stability interface, forming three protection circles (global, team, module). Regular case reviews, blue‑team exercises, and continuous metric analysis (MTTD, MTTR, interception rate) drive ongoing improvement.

Overall, the piece provides a detailed roadmap for building a resilient, observable, and automated search service, emphasizing rapid incident mitigation, systematic testing, and collaborative governance.