Mastering Traffic Spikes: Rate Limiting Strategies for Resilient Services

1 Introduction

In the "Microservice Series" we previously covered many concepts about rate limiting and circuit breaking. Service capacity is always limited—memory, CPU, thread count—and sudden traffic spikes require friendly rate‑limiting practices to avoid a full‑scale service avalanche.

Peak‑request scenarios mainly fall into two categories:

1.1 Sudden High Peaks Causing Service Avalanche

If your service encounters sustained, high‑frequency, unexpected traffic, you should check for erroneous calls, malicious attacks, or downstream logic issues. Such overload can increase latency, pile up requests, and trigger a cascade failure throughout the call chain.

1.2 Unexpected Traffic Floods (e.g., promotional events)

During large‑scale activities such as Double‑11 or 618, if you cannot accurately estimate the peak value and duration, the service still risks being overwhelmed. Only elastic scaling (dynamic auto‑scaling) can fully mitigate this risk, which will be discussed in the Cloud‑Native series.

In the example, normal traffic is 1500 QPS, the estimated model predicts 2600 QPS, but during the event traffic spikes to 10000 QPS, far exceeding server capacity, leading to latency, failures, request backlog, and possible avalanche.

2 Solutions

2.1 Cloud‑Native and Elastic Scaling

If your architecture is fully cloud‑native and robust, elastic scaling is the optimal solution. Platforms like Taobao, JD.com, and Baidu App use Kubernetes to adjust instance counts in real time based on CPU, memory, and traffic curves, scaling up during peaks and scaling down during idle periods.

2.2 Bottom‑Line Rate Limiting and Circuit Breaking

The most basic protection is to add a safeguard layer to prevent overload‑induced avalanches. Limiting traffic that exceeds expected capacity is essential, especially during high‑traffic events such as Double‑11, 618, flash sales, or auctions.

Service is loading, please wait.

Service/network error, please retry.

Oops, the service is busy, please try again later.

2.1 Application‑Level Solutions

2.1.1 Common Rate‑Limiting Algorithms

Counter Algorithm

The counter algorithm records the number of requests within a fixed time interval; when the interval expires, the count resets.

Fixed Window Algorithm (Sampling Time Window)

This adds the concept of a time window; the counter resets at each window boundary.

Sliding Window Algorithm (records each request timestamp)

The sliding window solves the fixed‑window edge problem, ensuring the threshold is never exceeded in any arbitrary interval.

Leaky Bucket Algorithm

Analogous to a sand‑hour, the outflow rate is constant, guaranteeing a steady processing rate for incoming requests.

Token Bucket Algorithm (steady token inflow)

Similar to the leaky bucket but with a constant token inflow; each request consumes a token, and only requests with a token are processed. When the bucket is full, extra tokens are discarded, and excess requests are rejected, achieving rate limiting.

2.1.2 Relevant Implementation Frameworks

Spring Cloud Hystrix

Sentinel (circuit‑breaker and degradation)

Google Guava RateLimiter

2.1.3 Actions When Rate‑Limiting Triggers

Fallback: return a fixed object or execute a predefined method.

<code>// Return a fixed object
{
  "timestamp": 1649756501928,
  "status": 429,
  "message": "Too Many Requests"
}

// Execute a fixed handling method
function fallBack(Context ctx) {
  // TODO: default handling logic
}
</code>

2.1.4 Web/Mobile/PC/3D UI Feedback

After receiving a fixed response, present user‑friendly messages such as:

Service is loading, please wait.

Service/network error, please retry.

Oops, the service is busy, please try again later.

2.2 Storage‑Layer Solutions

When Redis hot‑data receives massive concurrent requests (e.g., >10M), a cache miss can cause a stampede that overwhelms the database. Common mitigation strategies include:

Distributed lock – only one request accesses the DB, others wait, reducing DB load.

Queue‑based request execution – process requests sequentially to avoid DB overload.

Cache pre‑warming – ensure a portion of data is cached before heavy traffic.

Empty/default initial value – on first request, create an empty or default cache entry, query the DB, then update the cache; meanwhile, front‑end can show a friendly placeholder.

Local cache – store hot items in the web server’s memory in addition to Redis, reducing DB hits; combine with empty‑value or lock strategies for best results.

3 Summary

Whether at the application layer or the storage layer, the goal is to agree on a fallback rule with the front‑end, return default parameters or responses, and provide a user‑friendly experience that prevents the service from collapsing.