How We Built a Multi‑Layer Stability Framework for a High‑Traffic Transaction Platform

1. Introduction

The transaction middle‑platform is a core component of the WOS commerce operating system, linking product selection to order payment and handling pricing, discounts, delivery, and payment. It interacts with many domains (product, inventory, fulfillment, assets, orders, payment, merchants, users, promotions, marketing), making stability a major challenge.

2. Stability Considerations

2.1 Multi‑Dimensional Stability Characteristics

2.2 Multi‑Stage Stability Characteristics

Architecture design principles:

Closed for modification, open for extension, especially for public interfaces.

Prefer generic solutions, reduce specialization.

Avoid coupling; isolate business logic via switches.

Handle business exceptions carefully; do not implement compatibility logic without product confirmation.

Rate limiting, degradation, and circuit breaking are all mandatory; if circuit breaking is possible, do not limit, and vice‑versa.

3. Stability Dashboard

4. Transaction Stability Construction

4.1 Protection Strategies

Layer 1: Rate Limiting – Settlement (order confirmation & submission) and cart operations each have dedicated rate‑limit thresholds.

Layer 2: Circuit Breaking – Settlement does not use circuit breaking because it is the final step and cannot be degraded; cart operations have circuit‑break configurations for degradable downstream calls.

Layer 3: Timeout – Settlement and all cart‑related downstream calls are configured with timeouts between 1 s and 3 s.

Layer 4: Degradation – When downstream services are unavailable, traffic is cut off via switches to protect both downstream and the platform itself.

To implement these four layers we performed:

Mapping of strong and weak system dependencies together with product teams.

Analysis of module degradability and its impact on the system.

Preparation of business emergency and pre‑plan procedures for peak events.

4.2 Discovery Strategies

4.2.1 Runtime Discovery – Second‑Level Business Awareness Alert System

The transaction system depends on over 40 applications and 100+ interfaces. A second‑level business monitoring and alert system was built to instantly detect issues.

Monitoring includes:

Hardware metrics (CPU, I/O, network) via a generic monitoring platform.

Business‑level metrics such as code exceptions (NPE, OOM) and business anomalies (out‑of‑stock, resource freeze failures) that generic platforms cannot capture.

Features of the second‑level monitoring system

The Hawkeye SDK is integrated as a second‑party package, providing automatic system metric collection, custom event collection, second‑level aggregation, breakpoint‑resume for failed uploads, and minimal impact on business threads.

The Hawkeye server accepts HTTP and RPC reports, storing data in a TSDB.

4.2.2 Process System

Core idea: capturing the basic product and promotion data model enables coverage of fulfillment, inventory, assets, and merchant services.

4.2.3 Baseline Establishment

Historical load‑test data from Double 11 and regular traffic are stored online, and a tool for entering load‑test data models was built to facilitate later analysis.

4.2.4 Traffic Conversion

Traffic model + conversion model + data model → full‑link load testing → end‑to‑end stability.

Traffic model: distribution of traffic across promotion entry points (big promotion, detail page, activity page, cart).

Conversion model: loss ratios from activity page → detail page → cart → order confirmation → order submission → payment.

Data model: includes product, promotion, and user (regional) information.

4.2.5 Comparison Strategy

A online traffic replay comparison tool is used for large‑scale interface refactoring scenarios.

Basic Principles

External interface dependencies should have identical input parameters and theoretically identical output.

Core Thoughts

Record real request/response data of legacy online interfaces.

Asynchronously feed recorded data into new interfaces for input‑parameter comparison.

Benefits

Full scenario coverage.

Automated system, high efficiency.

Testing focuses only on changed parts; regression is automated.

Replay traffic does not generate actual load on downstream services.

5. Conclusion

As business complexity and traffic grow, maintaining system stability while supporting rapid iteration remains challenging; continuous refinement of deep, precise practices and architectural evolution is essential.