How Alibaba Prevents Release Failures in Billion‑Dollar Transactions

Introduction

Alibaba processes hundreds of billions of transactions daily, and any release failure can cause massive impact. To improve both release speed and stability, the team built a set of tools and an AI‑driven "unattended release" system that automatically detects and mitigates risks during deployment.

Pain Points of Online Release

Traditional releases rely on extensive manual monitoring of dashboards, logs, and alerts. Human‑in‑the‑loop monitoring is costly, error‑prone, and cannot keep up with the rapid data changes in large‑scale e‑commerce environments.

"Artificial Intelligence" Solution

Instead of pure manual observation, Alibaba introduced an AI‑assisted release assurance approach. Real‑time metrics (monitoring, release orders, machine health, GOC alerts) are collected and analyzed automatically to decide whether a release is safe.

Unattended Release

The core idea is to automate the entire monitoring and decision‑making process. The system continuously gathers key indicators, runs intelligent analysis, and instantly pauses a release if a potential fault is detected.

Two main capabilities: fault detection (identifying actual problems) and anomaly recommendation (highlighting suspicious but non‑critical issues for developers).

Unattended Release Details

On the release order detail page, a summary panel shows whether the current release has anomalies. If an issue is detected, the system automatically pauses the release and notifies developers via DingTalk.

Unattended Release Integration

Applications need to be integrated with the unattended release system. Most integrations are automatic; if an app cannot be auto‑connected, the system prompts the user to configure the necessary parameters.

Practical Case Study

A real‑world case shows a sudden spike in log exceptions during a release. The system highlighted the anomaly, developers confirmed the issue, and the release was rolled back, preventing a service outage.

Metrics and Evaluation

Two key metrics are used: recall (coverage of actual faults) and precision (rate of false positives). The system currently achieves ~90% recall and ~60% precision, balancing fault interception with acceptable false‑alarm levels.

Implementation Overview

The solution consists of three layers:

Release System Layer – the "HaiLang" platform handles release order submission, execution, and displays unattended release information.

Core Unattended Layer – collects real‑time data, runs analysis, and decides on pausing releases.

Offline Analysis Layer – trains algorithms, validates models, and provides feature libraries.

Data Collection & Processing

Time‑series metrics (CPU, memory, load, business KPIs, middleware QPS/RT, log exception counts) are collected from two groups of machines: the "release" group and a reference group. Only data around the release window is stored, then cleaned and aggregated for analysis.

Analysis Methods

Initial versions used rule‑based checks for obvious anomalies and a funnel‑style detection model for statistical deviations. Later, machine‑learning models were introduced to improve precision.

Machine‑Learning Pipeline

Offline learning builds a feature library from historical release data and feedback. During a release, the system flags suspicious metrics, compares them against the library, and decides whether to intervene. User feedback (e.g., rollbacks, explicit anomaly reports) is fed back into the training loop.

Continuous Improvement

By iteratively analyzing false positives/negatives, adjusting thresholds, and replaying historical releases through a dedicated replay system, the team steadily raised both recall and precision, reducing manual effort and increasing release reliability.