Stability Assurance Practices for Large‑Scale Promotional Events

Large promotional activities such as 618, Double‑11, and Double‑12 bring massive traffic spikes that test a company’s technical capabilities. In the context of high‑concurrency traffic, supporting business operations becomes a challenging task that puts high demands on system architecture and emergency safeguards.

On September 30 last year, Hello (哈啰) launched its first holiday carnival. The team carried out extensive stability‑protection work, and the event proceeded smoothly. This article summarizes the key stability‑guarantee measures and shares them with the community.

Differences between regular and promotional stability guarantees

Promotional periods are characterized by short duration, huge traffic, and rich marketing play‑styles. The main protection dimensions are capacity planning, pressure‑test rehearsals, emergency plans, and change control.

Overall promotional workflow

The workflow typically includes early‑stage planning, pressure‑test rehearsals, in‑event emergency response, and post‑event wrap‑up.

Organizational guarantee

A dedicated “promotion guarantee group” coordinates resources, makes decisions, and has the authority to pause non‑essential iterations during the event. Each business line appoints a promotion technical PM who is responsible for drafting a detailed guarantee plan based on business characteristics.

Target decomposition

Business goals (order volume, GMV, DAU, etc.) are translated into technical goals such as QPS and concurrent user count. This requires close communication with product to understand traffic sources, user paths, and system dependencies.

Pressure‑test rehearsal

After setting technical targets, pressure tests are conducted and the results are used to adjust the plan. Emergency scenarios are also rehearsed to validate the response procedures.

Change control

Changes are a major source of incidents; therefore, a pre‑defined change‑freeze schedule (application releases, configuration changes, operational changes) is established, and any necessary changes during the event are recorded for post‑mortem.

Internal gray release

Before full launch, an internal gray release is performed by adding internal users to a whitelist. Data isolation is crucial to avoid consuming real rewards during the test.

On‑call duty

A clear information‑sync mechanism is set up: a dedicated on‑call room, standardized IM group names, and inclusion of developers, product, operations, and customer service. Decision‑makers are also present to enable rapid issue resolution.

Detailed guarantee plan

The guarantee plan is a framework produced by the guarantee group; each technical PM refines it for their business line. Deliverables include link analysis, capacity water‑level tables, monitoring dashboards, emergency‑plan manuals, and coordination mechanisms with external partners.

Link analysis

Identify key entry points, downstream systems, and potential bottlenecks. Distinguish strong vs. weak dependencies and configure degradation, circuit‑breaker, and timeout settings accordingly. Detect hotspot traffic (e.g., push‑driven landing‑page spikes) for focused protection.

Capacity & water‑level analysis

Produce a QPS table for each system entry, indicating target QPS, current QPS, and whether scaling is required.

Monitoring & alerting

Ensure comprehensive monitoring coverage (infrastructure, middleware, application, traffic entry) with reasonable thresholds. Business‑level metrics (coupon exposure, redemption, order conversion) should be observable to track the promotion funnel.

Emergency plan

Prepare pre‑, during‑, and post‑event plans covering capacity expansion, rate limiting, cache warm‑up, and fallback strategies. Document triggers, actions, impact, owners, and execution speed.

Collaboration mechanism

Define on‑call personnel lists, communication channels, and information‑sync procedures across product, business, and customer‑service teams.

External partner guarantee

Coordinate with external APIs or services to monitor their latency and error rates, and establish fallback options if needed.

Different team focus

Front‑end business (e.g., bike sharing) emphasizes end‑to‑end user experience and strong downstream dependencies. Mid‑/back‑end services (e.g., payment, map, AI) must handle traffic aggregation from multiple upstreams and enforce isolation mechanisms.

Post‑event review

Compare actual system performance with pre‑event estimates and pressure‑test results. Analyze capacity utilization, emergency‑plan execution, and change‑freeze impacts to refine future guarantees.

Conclusion

Stability work covers many details; extracting common patterns into a methodology helps the organization grow. The 930 promotion demonstrated significant improvements in both business metrics and technical resilience, but continuous refinement remains essential.