High‑Availability Architecture for a Billion‑Scale Membership System

The membership system is a core service that connects all business order flows; any outage blocks user ordering across the entire company. After the merger of two platforms, the system must handle cross‑platform member queries and traffic that can exceed 20,000 TPS during peak periods.

ES High‑Availability Solution : A dual‑center master‑slave Elasticsearch cluster is deployed across two data centers (A and B). The primary cluster runs in A, the standby in B, and data is synchronized via MQ. In case of a primary failure, configuration switches redirect reads and writes to the standby, ensuring rapid failover. Additionally, a three‑cluster traffic‑isolation architecture separates critical order‑flow queries from high‑TPS marketing requests, preventing marketing spikes from affecting core transactions.

Deep ES Optimizations : Issues such as uneven shard distribution, oversized thread pools, large shard memory (up to 100 GB), and dual field mappings (text + keyword) were addressed. Thread‑pool sizes were limited to cpu_cores * 3 / 2 + 1 , shard memory capped at 50 GB, and filters were used instead of queries to avoid scoring overhead. Routing keys were added to direct queries to relevant shards, dramatically reducing CPU usage and query latency.

Redis Caching Scheme : Historically the system had no cache, but a sudden traffic surge during a ticket‑blind‑box event prompted the introduction of Redis. Near‑real‑time ES updates (≈1 s delay) caused cache inconsistency; the solution employs a 2‑second distributed lock before deleting the Redis entry, ensuring that stale data is not written back. Two‑center multi‑cluster Redis deployments write synchronously to both data centers, providing high availability and low latency reads.

High‑Availability Member Primary Database : The original SqlServer instance reached physical limits with over a billion records. A dual‑center MySQL partitioned cluster was built, splitting data into more than 1,000 shards (≈1 M rows each) with a 1‑master‑3‑slave topology (master in A, slaves in B). Full‑volume sync, incremental sync, and gray‑release traffic shifting were used to migrate without downtime, achieving >20 k TPS and ~10 ms average latency.

Abnormal Member Relationship Governance : Complex logic errors could bind a user’s APP account to another’s WeChat account, exposing orders across users. A thorough analysis identified such anomalies, and code‑level safeguards were added to prevent cross‑account data leakage.

Future Outlook – Fine‑Grained Flow‑Control and Degradation : Three levels of flow‑control are planned: hotspot throttling for abusive accounts, per‑calling‑account limits to guard against buggy client loops, and global limits to protect the system from traffic spikes beyond 30 k TPS. Degradation strategies include response‑time‑based circuit breaking and exception‑rate‑based fallback. Additionally, a systematic audit of all calling accounts will enable more precise control.