Design and Implementation of a High‑QPS Spring Festival Red‑Packet System Simulating 10 Billion Requests

The author was inspired by a 2015 article about building a reliable Spring Festival red‑packet system and decided to recreate a similar high‑load backend to validate the concepts. The goal was to support one million concurrent connections on a single machine, achieve a peak QPS of 60 000, and simulate the processing of 10 billion shake‑red‑packet requests.

Background

Key terms such as QPS (queries per second) and PPS (packets per second) are defined, and the business logic of shaking and sending red packets is explained.

Target Metrics

Based on public data, the author estimates that a single server should handle roughly 900 000 users, with a per‑server peak QPS of about 23 000 (derived from a total of 14 billion users across 638 servers). The practical target was set to 30 000–60 000 QPS.

Hardware & Software

Software stack: Golang 1.8r3, shell, Python; OS: Ubuntu 12.04 (server) and Debian 5.0 (client). Hardware: Dell R2950 8‑core server with 16 GB RAM, and 17 ESXi 5.0 VMs (4 CPU/5 GB RAM each) acting as clients.

Implementation

The system uses a set‑based architecture where one million connections are divided into multiple independent SETs, each managing a few thousand connections. This design enables horizontal scaling by adding more SETs or servers.

Key techniques include:

Limiting goroutine count by assigning a single goroutine per connection and a dedicated goroutine per SET.

Using NTP‑synchronized timestamps for client request coordination, with a simple algorithm that groups users to evenly distribute requests.

Lock reduction by partitioning users into buckets and optionally employing the Disruptor queue for higher performance.

Monitoring is performed via a custom Python script that leverages ethtool to track network packet statistics, and a lightweight monitoring service collects per‑client counters.

Practice

The experiment proceeds in three stages:

Start server and monitoring services, then launch 17 client VMs to establish one million connections.

Set client QPS to 30 000 and verify request rates via network and monitoring dashboards.

Increase client QPS to 60 000 and repeat the verification.

Command used to list connections:

Alias ss2=ss –ant | grep 1025 | grep EST | awk -F: "{print $8}" | sort | uniq -c

Data Analysis

Graphs (client QPS, server QPS, red‑packet generation, and shake‑red‑packet success rates) show stable performance at 30 000 QPS and increased variance at 60 000 QPS due to network jitter and goroutine scheduling overhead.

Profiling indicates occasional GC pauses >10 ms, acceptable given the seven‑year‑old hardware.

Conclusion

The prototype meets the design goals: a single‑machine backend that supports one million users and up to 60 000 QPS, capable of processing 10 billion shake‑red‑packet requests in under 12 minutes when scaled across 600 servers. Differences from a production system (complex protocols, payment integration, detailed logging, security, hot‑updates) are acknowledged.